System and method for automated model calibration, sensitivity analysis, and optimization

ABSTRACT

A computer-implemented interface apparatus for automated calibration can include an architecture for automatically managing interchangeable input parameters, interchangeable output objective functions, and interchangeable optimization methods. Prior to calibration, this architecture allows a user to quickly and easily eliminate the vast majority of input values and combinations, thus drastically simplifying the process of calibration (via simulation-based optimization). The interface apparatus can be used to provide an efficient and practical self-calibration method for computer models, having any number of input and output parameters. Users can adjust the selection of parameters to affect, and in some cases fully control, required computer run times for automated calibration.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a divisional application of U.S. applicationSer. No. 14/446,783, filed Jul. 30, 2014, which claims the benefit ofU.S. Provisional Application Ser. No. 61/859,819, filed Jul. 30, 2013,the disclosures of each of which are hereby incorporated by reference intheir entirety, including any figures, tables, or drawings.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF INVENTION

The use of computer implemented models to evaluate complicated systemshas become increasingly popular. A well-designed, adequately validatedmodel can be used to understand current conditions and predict futuretrends. The ability to accurately model multi-variable systems can aidin optimizing current resources and allow for future planning.

One area in which computer models have become increasingly important isin the area of vehicle traffic control and road maintenance. Thecontinued increase in population means that the number of vehiclestraversing the roadways has also increased and will continue toincrease. As a result, it has become more important for large and smallmunicipalities to be able to improve traffic safety and determine howlimited resources for road construction and maintenance can be optimallyallocated in their areas. Software modeling applications for simulatingtraffic and road conditions are becoming more commonly used as a toolfor analyzing traffic operations.

This increase in their importance has led to computer programs fortraffic simulation becoming increasingly more advanced and sophisticatedover the past two decades. Usage of traffic simulation has increasedsignificantly; and this high-fidelity modeling, along with movingvehicle animation, has allowed important transportation decisions to bemade with better confidence. During this time, traffic engineers havetypically been encouraged to embrace the process of calibration, inwhich steps are taken to reconcile simulated and field-observed trafficperformance.

According to international surveys, top experts, and conventionalwisdom, existing (non-automated) methods of calibration have beendifficult and/or inadequate. Consulting engineers and DOT personnel haveexpressed strong interest in making calibration faster, cheaper, easier,and requiring less engineering expertise. Some users of simulations havebeen unwilling to perform any amount of calibration; frequently citinglabor-intensive data collection procedures, or a lack of coherentprocedures and guidelines. Some simulation users have also tried toapply procedures and guidelines that exist in the literature; but havefound that these guidelines are difficult to apply, or that theguidelines are a poor fit for their specific type of simulationanalysis. Finally, some simulation users believe that they have somewhatmastered the process of calibration; but that the amount of engineeringexpertise required to achieve this mastery could be measured in decades,or that successful execution of calibration for a new project couldrequire many weeks of hard work.

There has been a significant amount of research in the area of automatedcalibration techniques, for traffic simulation. However, many of theseresearch projects and papers have not provided the level of flexibilityand practicality that are typically required by real-world engineers.There is a need for a calibration method that can increase theeffectiveness of computer models. Such a calibration method will ideallybe simple to implement, provide sufficient customization to accommodatedifferent user needs, and increase the efficiency of computer models.

BRIEF SUMMARY

The embodiments of the subject invention provide a method forself-calibration of complex computer models that can potentially utilizemultiple simulation data parameters. More specifically, embodiments ofthe subject invention provide architecture for software-assistedcalibration that can be used with a simulation-based optimization (SO)family of simulation models. In simulation-based optimization, thenumeric discrepancy between simulated and field-measured results can bean objective function to be minimized. Certain implementations of thesubject invention provide an efficient and accurate method forself-calibrating traffic simulation models. One or more methods of thesubject invention described herein can:

-   -   1. Make calibration easier, and reduce the amount of engineering        expertise required for calibration.    -   2. Reduce the amount of time and money typically required for        calibration.    -   3. Improve the standard of accuracy for future simulation        analyses.    -   4. Increase uniformity of simulation practices, which could lead        to increased acceptance of simulation for traffic analysis and        other types of system models.    -   5. Provide a design blueprint for other simulation developers to        utilize.    -   6. Provide material for simulation guidelines.

Embodiments of the subject invention provide architecture forsoftware-assisted calibration, within a simulation-based optimization(SO) family of methods. The new architecture can use a database ofreal-life based default values for various input parameters, topre-define a narrow set of trial values to be used during optimization.The architecture can also allow engineers to prioritize input and outputparameters, and specify a tolerable computer run time, prior toinitiating the SO-based calibration process. The use of a “directedbrute force” (DBF) search process algorithm can make the architectureflexible and practical, for real-world use. These same features can alsobe extended to provide an enhanced platform for sensitivity analysis,and optimization. The acronym term “SASCO” (Sensitivity Analysis,Self-Calibration, and Optimization) refers to a database-centricframework embodiment that can support any one of these three analysistypes.

SASCO can be classified as belonging to the simulation-basedoptimization (SO) family of methods Under SO, numerous inputcombinations are simulated for the purpose of minimizing or maximizingan objective function. When the goal of SO is calibrating a simulationmodel, minimizing the difference between simulated and field-measuredoutputs is sometimes the objective function. In addition to theobjective function, another important aspect of SO is the searchingmethod. Intelligent searching methods are designed to obtain the bestpossible solutions in the shortest amount of time, or using the smallestnumber of trials. Intelligent searching methods would not be needed ifcomputers were infinitely fast; optimum solutions could be located byexhaustive (“brute force”) searching, which would simulate everypossible combination of inputs. But given the speed of modern computers,brute force optimization is not practical for computationally expensivetraffic simulations.

Embodiments of the subject invention can also incorporate sensitivityanalysis for one or more input parameters. Sensitivity analysis is oftenused in systems modeling to study the uncertainty that can affect theoutput of a mathematical-based model or system and that can beapportioned to different sources of uncertainty in its inputs.Sensitivity analysis can be useful for testing the robustness of amodeling system in the face of uncertainty in input parameters. It canalso lend a greater understanding of the relationships between input andoutput variables, which can help to reduce uncertainty in the modelinputs.

Optimization of an input parameter is precisely what it implies, whichis finding the best possible value for a particular input parameterwithin a range of possibilities. However, optimization of a model withnumerous possible input parameters can be time consuming. As a result,model systems often focus on optimization of variables that are mostoften associated with the particular output parameters being reviewed.This can obtain results faster; but, it also reduces the possibility ofdiscovering unknown input variables that affect those same outputparameters. Embodiments of the subject invention can increase the numberof optimized input parameters to improve output parameter results.

As with any computer software simulation method, there can be any of avariety of limitations that affect the accuracy of the calibrationmethod of the subject invention. Largely due to computer speedlimitations, it is believed that automated calibration processes cannotfully replace user judgment, user expertise, or manual (non-automated)calibration. Automated self-calibration also cannot defend againstfundamental (volume, timing, laneage) input data errors, simulationsoftware bugs, simulation software limitations, inconsistent performancemeasure definitions, and other limitations and factors understood bythose with skill in the art. However, if properly designed and used, theautomated process of the subject invention can reduce the amount of timeand expertise typically required for calibration. Simulated performancemeasures can be made to match field-measured performance measures byseveral methods, including, for example:

1. Correction of fundamental input data (e.g., volume, timing, laneage)errors

2. Reconciliation of inconsistent performance measure definitions*

3. Collection of more accurate field-measured values

4. Elimination of simulation software limitations*

5. Correction of simulation software bugs*

6. Manual (non-automated) calibration

7. Automated self-calibration

(*usually handled by the simulation software developers)

Embodiments of the subject invention provide several unique advantagesover the currently used calibration methods. One embodiment allows auser to select any number of field-measured output parameter values andany number of items from an available database of real-life baseddefault values for various input parameters. A further embodimentutilizes a “directed brute force” methodology to provide at least threelevels of data optimization (quick, medium, and thorough). With thedirected brute force method, a user is able to selectively calibrateindividual parameters by determining the level of calibrationthoroughness utilized for each input parameter. Other algorithms can beincorporated as well that are capable of increasing the sensitivity andoptimizing results. Implementation of the calibration method allows auser to determine and select data for each chosen parameter thateffectively simulates real-world observations. In a further embodiment,calibration comprises archiving user-selected data to be utilized forfuture simulations, to obtain more accurate results.

Further implementations display an estimate of the amount of timerequired to conduct the number of simulations necessary to achieve thelevels of data optimization that have been selected (i.e., quick,medium, thorough). This, in turn, allows a user to select and adjust thelevels to obtain a tolerable number of runs or simulations, and, thus, atolerable run time, or simulation time, in advance.

Embodiments of the subject invention also provide flexibility to theself-calibration technology. In one embodiment, various selected inputparameters can be weighted prior to simulation to further adjustrun-time and the accuracy of resulting output values. In anotherembodiment, a random number seeding method can be used to examine afuller range of possible outcomes. Another embodiment allows users tocustomize a database of candidate input parameter values. This allows auser to eliminate values that are considered to be outliers ornon-applicable for a particular simulation.

Advantageously, embodiments of the calibration method of the subjectinvention are easy to use. In one embodiment, users can automaticallyload/display all outputs from an original run and, in a furtherembodiment, available field-measured values can be entered for anyparameter. Other embodiments can include various display options. Oneembodiment displays the original average percent difference between realworld data and model coefficient values. Another embodiment displaysresults during a run or simulation process, and allows a user to abort arun if necessary. For example, if it becomes apparent that thesimulation is inaccurate as simulation results are displayed, it may bedetermined that certain parameters need to be adjusted and continuingthe current simulation process is unnecessary. Embodiments of thesubject invention provide the advantage of permitting a user the optionof aborting a simulation process during mid-operation to avoid wastingtime and resources. A still further embodiment can display standarddeviations between candidate (trial) calibration parameters, revealingmodel sensitivity to the calibrated input parameter values.

Self-calibration features in the embodiments of the subject inventionadvantageously maximize practicality, flexibility, and user-friendlinessof computer model calibration. In particular, the invention has beendemonstrated effective in the arena of traffic simulation models. Theimplemented methodologies allow users to quickly and easily select a setof input and output parameters for calibration. This methodology alsoallows users to prioritize selected specific input and outputparameters, and to further specify a tolerable computer run-time, priorto initiating the self-calibration process. The “directed brute force”search process utilized in implementations herein further allow themethodology to be sufficiently flexible and practical for real-worlduse.

BRIEF DESCRIPTION OF DRAWINGS

In order that a more precise understanding of the above recitedinvention can be obtained, a more particular description of theinvention briefly described above will be rendered by reference tospecific embodiments thereof that are illustrated in the appendeddrawings. Thus, understanding that these drawings depict only typicalembodiments of the invention and are not therefore to be considered aslimiting in scope, the invention will be described and explained withadditional specificity and detail through the use of the accompanyingdrawings in which:

FIG. 1A is a flowchart illustrating the basic steps of one embodiment ofthe invention.

FIG. 1B is a flowchart that illustrates a comparison of the features ofthe calibration method of the subject invention with other methods ofcalibration known in the art.

FIG. 1C is a diagram illustrating a typical operating environment of asimulation model.

FIG. 2 is a flowchart that illustrates a method of self-calibration,according to one implementation of the subject invention.

FIG. 3 shows one implementation of an Output Parameters interfacescreen.

FIG. 4 shows one implementation of an Input Parameters and Run Statusinterface screen.

FIG. 5 shows one implementation of a pop-up dialog box used to calibratea local subset of input parameter values within a model, as opposed toglobally adjusting all values of a given input parameter. Here it isshown that CORSIM input parameters Mean Queue Discharge Headway(NETSIM), and Car Following Sensitivity Multiplier (FRESIM), can becalibrated in specifically chosen areas of the traffic network.

FIG. 6 shows an Input Parameters and Run Status interface screen at theconclusion of a run.

FIG. 7 shows a dialog box used to import a specific trial dataset.

FIG. 8 shows a dialog box listing a database of values to be used withinput parameters for self-calibration.

FIG. 9 shows a pop-up dialog box that allows a user to input whetherexisting calibration settings shown on the interface screen can beimported for the selected parameters.

FIG. 10 shows an example of simple link-node geometry for a simplesurface-street section with two signalized intersections separated by1800 feet. Example 5 describes the calibration of this particularexample.

FIG. 11 shows an input parameter interface screen with a dialog box toselect loading of one set of candidate values for a particular trial,i.e., the 34.trf trial.

FIG. 12 shows an example of link-node geometry for an extended freewayfacility with eleven segments, and five time periods. Example 5describes a method for calibrating a model of this example.

FIG. 13 is a flowchart illustrating one embodiment of a SensitivityAnalysis, Self-Calibration, and Optimization (SASCO) system according tothe subject invention.

FIG. 14 is an example of a user-interface screen design for selectingcalibration input parameters to be calibrated by the embodiments of thesubject invention.

FIG. 15 illustrates an example of a user-interface screen listing avariety of selectable output parameters from which a user can chose oneor more of interest for a particular calibration run.

FIG. 16 illustrates an example of database values selected for quick,medium, or thorough calibration, where successive levels entail agreater number of simulations.

FIG. 17 illustrates one embodiment of a user interface screen forselecting output parameters.

FIG. 18 illustrates one embodiment of a user interface screen forreviewing results of sensitivity analyses.

FIG. 19 illustrates one embodiment of an output parameters screen,according to the subject invention.

FIG. 20 illustrates one embodiment of an input parameters and run Statusscreen, according to the subject invention.

FIG. 21 shows a list of a certain number of input parameter names thatbegin with the phrase “Link-Specific”, as described in Example 5.

FIG. 22 illustrates one embodiment of a pop-up dialog box to allowautomated calibration on some links and not others, as described inExample 5.

FIG. 23 illustrates one embodiment of an input parameters and run statusscreen at the conclusion of a run, as described in Example 5.

FIG. 24 illustrates one embodiment of a pop-up dialog box for selectingwhether to import calibration settings for the selected data set.

FIG. 25 illustrates an interface screen of some of database values forinput parameters. In this embodiment, the database contains multiplepreset input parameters within a few dozen text files.

FIG. 26 illustrates an example of the types of database values that canbe utilized when certain input parameters are selected for calibration.

FIG. 27 illustrates one embodiment of an interface screen with anexample of the Sensitivity Analysis (SA) results conducted for alink-specific left-turn pocket length simulation.

FIG. 28 illustrates an example list of the contents of a database textfile, according to the subject invention.

FIG. 29 illustrates one embodiment of an interface screen that shows adata file called “34.trf” contains a % Difference value that is closestto the Mean Value for all 100 trial datasets.

FIG. 30 illustrates the link-node geometry for a “Surface and FreewayDemo” example network, from Traffic Software Integrated System (TSIS),specifying a single 15-minute time period, with 9 surveillance detectorsavailable on three freeway links (2->51, 510->52, 57->56).

FIG. 31 illustrates one embodiment of a dialog box that can be used forlink-specific calibration, according to the subject invention.

FIG. 32 illustrates the link-node geometry for an “Actuated ControlDemo” example network, from TSIS that specifies two 5-minute timeperiods, as described in Example. 5.

FIG. 33 illustrates one embodiment of an interface screen used forsetting the measured SpeedAverage (Cumulative, Global) equal to itssimulated value (17.64), as described in Example 5.

FIG. 34 illustrates one embodiment of an interface screen for displayinga table of sensitivity analysis (SA) results and shows that trafficnetwork performance, in this example, generally deteriorates as thepercentage of trucks increases.

FIG. 35 illustrates one embodiment of an interface screen for settingthe measured Delay Control Per Vehicle (Cumulative, Link-Specific) equalto its simulated value (32.01), on link 1-->2, as described in Example5.

FIG. 36 illustrates one embodiment of a dialog box for selectingspecific nodes for calibration, as described in Example 5.

FIG. 37 illustrates one embodiment of an interface screen for displayinga table of optimization results. The example in this figure shows thatcontrol delay is minimized at a yield point of 25 seconds, as describedin Example 5.

FIG. 38 is an example where tabular outputs for a particular data sethave been imported into a spreadsheet file, so that all desired outputscan be simultaneously reviewed.

FIG. 39 illustrates one embodiment of slider controls. In this example,the slider controls are used to dynamically adjust the range limits fortimes and includes checkboxes to select cost ranges.

DETAILED DISCLOSURE

The subject disclosure describes embodiments of a calibration method forcomputer simulation programs. More specifically, the subject disclosureprovides methods for calibrating simulation programs having numerouspossible input and output variables. Methods are described that providethe ability to select the number and type of variables utilized by thesimulation program, so as to control the run-time of the software andthe accuracy and relevance of the information output. Specificembodiments are database-centric, such that they utilize a database ofpre-determined values use with input parameters to conduct one or moresimulations.

In general, methods of the subject invention can be employed with anycomputer model that utilizes multiple input and output variables.Certain embodiments of the subject invention conduct simulations byemploying a database populated with pre-determined values for eachselectable input variable. Alternative embodiments can employ a methodwherein a large range of values is available and presented to the userfrom which a constrained sub-range of values can be dynamicallyselected. Further embodiments allow a user to selected the stepwisevalues to be used within the constrained sub-range. Embodiments ofsubject invention, which can employ specific, selectable, algorithmsduring simulations, provide a unique advantage of being able to predictthe run-time of a simulation scenario, based upon the number of selectedinput parameters, and based on the chosen calibration thoroughness ofeach selected input parameter. This predicted run-time can be providedto a user for determining if the run-time, or the time required toexecute a selected set of simulation parameters, is acceptable. If therun-time is unacceptable a user can alter the number of selected inputparameters, and/or the calibration thoroughness of any input parameters,so as to adjust the run-time.

The following description is directed to an implementation that isparticularly useful in the field of traffic and road conditionsimulations. However, a person with skill in the art will be able torecognize numerous other uses that would be applicable to the devicesand methods of the subject invention. Thus, while the subjectapplication describes, and many of the terms herein relate to, a use forcalibration of traffic simulation software, other modifications apparentto a person with skill in the art and having benefit of the subjectdisclosure are contemplated to be within the scope of the presentinvention.

With reference to FIGS. 1B, 2 and 13, it can be seen that the method ofthe subject invention can be used to calibrate any of a variety ofsoftware implemented mathematical models. FIG. 1B illustrates a generaloutline of the operations conducted with typical mathematical models.FIGS. 1A and 1B also show various steps conducted by certainimplementations of the calibration method of the subject invention andhow they compare to other known calibration methods. It can be seen inFIG. 1B that the calibration method of the subject invention providesability and flexibility previously unavailable in model calibrationmethods.

Mathematical models can include, but are not limited to, simulation ofbehaviors of business, economic, social, biological, physical, mental,and chemical systems, or combinations thereof. FIGS. 2 and 13 illustratethe basic steps utilized by embodiments of the calibration method of thesubject invention and how they can be applied to example models thatsimulate traffic, weather and personal health.

In one implementation, the calibration method of the subject inventionis a software program that can be integrated with a mathematicalmodeling program. As shown in FIG. 2, the initial step can be for a userto perform at least one simulation run with the model, to generate thesimulation outputs necessary prior to calibration. For traffic modelingimplementations, calibrations for time period-specific outputs, inaddition to, or instead of, cumulative outputs can require generation ofadditional files specific to the model being calibrated.

In one embodiment, the model 10 is implemented by selecting inputparameters 12 that can be relevant to the output parameters 14 ofinterest, or are otherwise utilized by the modeling program to generateoutput parameters. FIG. 2 is a flow chart illustrating the general stepsfor three examples of mathematical models used to simulate traffic,weather and personal health parameters. FIG. 2 also describes examplesof the types of input parameters 12 that might be chosen to obtainappropriate output parameters 14.

In a particular embodiment, a model that simulates traffic patterns canhave specific input parameters, such as those shown, for example, inFIG. 4. FIG. 3 illustrates examples of some output parameters 14 thatare typically generated by traffic simulation. A preliminary simulationrun can be performed, to generate an initial set of simulated outputmeasurements 16.

The output measurements 16 are then compared to “ground truth” values,which can be field measured or empirical values, user estimated values,or values obtained from another analysis tool, such as, for thisexample, a separate traffic analysis tool. In one embodiment, the outputmeasurements 16 are then compared to empirical or real-world measuredobservations 18 for the same parameters. FIG. 3 illustrates an exampleof a computer interface screen showing a list of output parameters 14selected for a traffic simulation model and the simulated outputmeasurements 16 along with the empirical measurements 18 for theselected parameters. In a further embodiment, also shown in FIG. 3, thesimulated output measurements 16 are compared to the empiricalmeasurements 18 and a percent difference 19 between the two values isdisplayed. The preliminary simulation can indicate where the modeldeviates from real-world observations and provide information as towhich parameters need to be calibrated.

By default, all output parameters have an equal effect on the overallpercentage difference between modeled and measured performance. In otherwords, they are all given the same weight, i.e., 100%, by default. In afurther embodiment, the output parameters can be weighted, to reflecttheir desired influence upon the calibration process. FIG. 3 shows a “%weighted column” 20 in which parameter weights can be viewed and/oradjusted.

Once an initial simulation has been conducted, selected input parameterscan be calibrated. In one embodiment, a user determines which inputparameters to calibrate, in order to minimize the % Difference 19, whichis the difference between the model simulated results and the modelgenerated value based upon ground truth values, like empirical data.FIG. 14 illustrates an example of an input screen that allows a user toselect multiple input parameters to be calibrated. If certain inputparameters only require calibration within subsets of the overallsystem, a user can also define 22 those subsets. FIG. 2 illustratesexamples of model-simulated output parameters having a % Difference 19of 7% when compared to actual, empirical, measured output parameters.FIG. 15 illustrates an example of a user-interface screen listing avariety of selectable output parameters 14 from which a user can choseone or more of interest for a particular calibration run.

Once a user has determined which simulated output parameters need to becorrected for accuracy, i.e., calibrated, the appropriate inputparameters that affect the desired output parameters can be selected forcalibration. In the example shown in FIG. 2, a user can choose tocalibrate specific input parameters, such as, for example, driveraggressiveness, solar activity or meat consumption, within each,respective, model. FIG. 14 illustrates an embodiment of an input userinterface screen that lists a variety of selectable input parameters 12from which one or more can be chosen for calibration. A user interface(U/I) can be part of an apparatus that includes instructions stored on acomputer readable storage media. Utilization of a user interface, asemployed with the embodiments of the subject invention, can be conductedwith any of one or more various input devices and output devices, suchas, for example, touchscreens, keyboards, mice, external displays,speakers, and screens, and other devices suitable for inputting oroutputting information as described herein, and combinations thereof.

Embodiments of the subject invention can also be used to facilitatesensitivity analysis (SA) for specific input parameters. SA has longbeen relied upon by professionals as a way to gain a betterunderstanding of a system model, and to identify those input parametersthat have the strongest impact on model results. Sensitivity analysiscan allow input parameters not normally associated with calibration(e.g., desired free-flow speed, entry node volume, percent trucks, etc.)to be examined, in an automated manner, to determine if they have anyspecific impact on output results. The input parameters most commonlyassociated with calibration can also be quickly analyzed via automatedSA as well.

With regard to output parameters generated by embodiments of the subjectinvention, a user can still select desired performance measures on theOutput Parameters screen. In one embodiment, measured values that matchthe simulated values could be used. This can allow the percentdifference between simulated and empirical values or ground truthvalues, prior to sensitivity analysis, to be displayed as 0.0 forexisting conditions. This can provide the advantage of more clearlyunderstanding the impact of a selected input parameter on the modelresults.

In one embodiment, the database 60 can be customized with specific inputparameters. This can be accomplished by modifying, deleting, or addingto existing input parameters values. In a specific embodiment, acandidate data set 40, such as, for example, in the form of a text file,as shown in FIG. 8, containing values for a particular input parametercan be added to the database 60. Once this text file has been added tothe database, that particular input parameter will then be available toselect on the Input Parameters screen 15. After selecting that inputparameter, sensitivity analysis can be performed by completing acalibration cycle. In further embodiment, the sensitivity analysisresults can be viewed in a tabular format by going to the associatedOutput Parameters screen 17. In a further embodiment, the outputparameters screen can include sensitivity analysis results selector 68that, when invoked, will display the results of the sensitivityanalysis. FIG. 18 illustrate a non-limiting example where a sensitivityanalysis results selector 68 is a button on the output parametersscreen. In a further embodiment, the sensitivity analysis resultsselector only becomes available after trial datasets have been generatedon the Input Parameters screen. Advantageously, the sensitivity analysisfeature of the subject invention allows a user to automatically contrastany of a variety of simulation input and outputs.

In one embodiment of the subject invention, a “directed brute force”(DBF) calibration method is utilized to calibrate specific inputvariables. A regular brute force (BF) calibration method conductsexhaustive trials, wherein it simulates every mathematically possiblecombination of input parameters. Such a method could be used withdifferent models to determine which input parameter combinations(contained within candidate data sets 40) provide the most accuratesimulation, when compared to measured output parameters. Those candidatedata sets that most closely match the ground truth values, such as, forexample, empirical results, can be isolated and used to conduct futuresimulations, to further refine or calibrate the model, or be used forpredictive analysis. In effect, DBF calibration culls through allallowable input parameter values to determine which combination(s) ofselected input variables, and their associated data sets, providesimulated output parameters that most closely match the measured outputparameters, which are based empirical data or other types of groundtruth values. This method has the advantage of searching completelythrough input parameter values in a regular and reproducible manner. Aswill be discussed below, this can be advantageous in predicting arun-time value for any given simulation. However, computer run-timerequirements of this method can be overwhelmingly impractical, for avast majority of real-world cases. Other search methods (e.g., geneticalgorithms, downhill simplex, latin hypercube, hill-climb, gradient,Simultaneous Perturbation Stochastic Approximation (SPSA)) can requirefewer trials than regular brute force searching, but may not provide theflexibility and practicality needed for most situations and the inherenttime requirements can inhibit public acceptance of such calibrationmethods.

Implementations of the “directed” brute force (DBF) calibration methodof the subject invention overcome these disadvantages by allowing a userto select the amount of calibration that will be conducted for eachvariable, thus allowing total control over the number of trials 5 andthe accuracy of calibration performed on any selected input parameter.In one implementation, after an initial simulation has been conducted,as described above, a user will be presented with an output parametercalibration results screen 17 which displays information about selectedoutput variables, such as shown, for example, in FIG. 3, that can beused to determine which of the selected output variables need to becalibrated. In one embodiment, an input parameter calibration selectionscreen 15 is provided, non-limiting examples of which are shown in FIGS.4 and 14, so that a user can select for each input variable to becalibrated, the level 30 of brute force calibration desired forindividual input variables, such as, for example, quick, medium, orthorough. This ability to select the level of calibration thoroughnessallows a user to “direct” the amount of calibration desired for eachparameter. In one embodiment, the selected level 30 can determine thenumber of trials 5 or searches conducted, using the pre-determinedvalues provided in the database 60, that the calibration method willconduct. In one implementation, a user can select from quick, medium, orthorough calibration levels. FIG. 16 illustrates an example of databasevalues selected for quick, medium, or thorough calibration, wheresuccessive levels entail a greater number of simulations. Each of theselevels can be set to conduct any specific number of searches through theinput parameter values within the database 60. In one embodiment, thecalibration thoroughness is predetermined, such that the method willonly conduct a designated number of trials for that level. For example,a low level search can conduct and analyze on just a few trials 5, amedium level can conduct and analyze more trials and a thorough levelcan conduct and analyze still more trials. FIG. 13 illustrates anexample of a pre-run input parameter calibration screen 15, which can beused to initially select the input parameters to be calibrated. It iswithin the skill of a person trained in the art to determine the numberof trials that each level will conduct. Such variations which providethe same function, in substantially the same way, with substantially thesame result, are within the scope of the subject invention.

In one embodiment, shown, for example, in FIGS. 4 and 6, quickcalibration 32 will conduct 3 trials, medium calibration 34 will conduct5 trials and thorough calibration 36 will conduct 10 trials. In afurther embodiment, the same or a different level of calibration can beselected from each input variable. For example, quick calibration 32 (3trials) can be selected for a first input parameter, medium calibration34 (5 trials) can be selected on a second input parameter, and thoroughcalibration 36 (10 trials) can be selected for a third input parameter.With this calibration strategy, there would be 150 different trials 5that product 150 different candidate data sets 40 derived from thedatabase 60 of trial values for each input parameter, i.e., 3 trials*5trials*10 trials=150 candidate data sets. In a further embodiment, trialdata sets are automatically archived by the model for future use. Thiscan be accomplished by a variety of techniques, including, but notlimited to, marking the data within each respective data file,extracting it or copying it to another file, saving it within anotherarea of the same file, or saving virtual data sets within memory. Aperson with skill in the art would be able to determine how the trialdata can be saved for future use.

In a further embodiment, random number seed generators are madeavailable as input parameters subject to calibration, for the purpose ofaccommodating stochastic models. Stochastic models have the commoncharacteristic of producing different output results when the randomnumber seed is changed. Repeated analysis of a data set will produceidentical results if the random number seed is not changed. Randomnumber seed generators are often utilized in software and are well-knownin the art. One embodiment of the subject invention utilizes at leastone seed parameter default list 35. In a specific implementation for atraffic simulation model, three seed parameter default lists 35 areutilized. FIG. 6 shows a non-limiting example of a post-run inputparameter calibration screen 21 that includes the ability to select anEntry Headway Seed; Traffic Stream Seed and Traffic Choice Seed filesthat contain default seed parameters that can be used with trafficsimulation model embodiments of the subject invention.

In a further embodiment, the seed parameters default lists can be usedfor an initial self-calibration run to determine if individual trial 5results may be “outliers”. This can provide better statistical certaintyand a fuller range of possible outcomes. One embodiment for utilizingthe random number parameters is illustrated in Example 1, below.

Traffic simulators often contain random number seed (RNS) data entry, toanalyze stochastic effects. Changing the RNS can influence driver suchparameters as driver aggressiveness, driver decisions, headways betweenvehicles, etc. When only one simulation is conducted, the results can bemisinterpreted as being typical, average real-world results. But, whenanalyzing unstable environments or situations, such as, for example,traffic conditions, numerous simulations (with different RNS) may beneeded to provide sufficient confidence in the final results. This canpresent a dilemma for automated, simulation-based optimizationcalibration. For example, if 10 simulations (with different RNS) wereneeded for each combination of calibration inputs, this woulddrastically inflate computer run times. The embodiments of the subjectinvention provide some assistance in addressing uncertainty andrandomness. At the top of FIG. 14, it can be seen that RNS inputs(Traffic Stream Seed, Traffic Choice Seed, etc.) are available forselection, similar to the calibration inputs. These RNS inputs can belocated at the top of the list to encourage end-users to select thesespecific types of parameters to analyze randomness. This allowssimultaneous optimization of RNS and calibration inputs; but run timescould be high, and results difficult to interpret.

In a further embodiment, to manage randomness when run times are high,“pre- and post-” analysis is one option. Prior to conducting acalibration run, a pre-calibration run could be performed to “calibrate”and import a RNS producing the most “average” results. These RNS couldthen remain in effect during the standard calibration run. Afterimporting the optimized inputs, a post-calibration run could beperforated, to determine whether the solution was stable for differentRNS. Pre-calibration stochastic analysis would increase odds of a stablefinal solution; but if unstable, the “average” RNS could be imported atthis time. The sequential technique of “pre- and post-” stochasticanalysis is not a perfect strategy because randomness is ignored duringa calibration run. Another option can be to perform multipleoptimizations, but manually change the RNS before each calibration run.For example, after five optimizations one could assess variance of thefive final objective function values, and then import calibratedsettings that produced the median final objective function value. Theimportant point is that embodiments of the subject invention allowsend-users to control how much stochastic analysis is performed; whetherthat involves sequential optimizations, or whether that involves reducedlevels (Quick/Medium/Thorough) of RNS replication.

The ability to select levels of calibration thoroughness, to beconducted within the database 60 for each input parameter,advantageously provides a user with control over the level ofcalibration necessary or desired for a particular simulation scenario.By selecting the calibration level for each input parameter a user canalso control the amount of time to be dedicated to each automatedcalibration scenario, as described in FIG. 2. In one embodiment, themodel can generate an estimate of the time required to generate resultsfor a particular input scenario. In a particular embodiment, the timerequired to run the initial simulation, conducted prior to calibration,is multiplied by the number of trials that would be necessary based uponthe number of input parameters and calibration levels 30 selected. Thus,in the example provided above, the initial simulation time can bemultiplied by 150 to obtain an estimate of the amount of time it willtake to run the selected simulation scenario with 150 trials.

In a further embodiment, a calibration method that employs a DBFalgorithm can generate the time estimate prior to the start ofself-calibration. This provides a user with the ability to alter theself-calibration scenario, such as, for example, by altering the searchlevels selected for each parameter, so as to change the time estimate.Thus, if a user can tolerate self-calibration with a longer timeestimate, changes can be made to the search level(s) selected for eachinput parameter 12, perhaps selecting thorough calibration 36 for moreof the parameters. Likewise, if the time estimate for a self-calibrationis too long, calibration levels and/or number of selected inputparameters can be adjusted to reduce the time estimate. FIGS. 4 and 6illustrate embodiments of interface screens on which a time estimate 38,generated by the calibration method, is provided. FIG. 4 illustrates anembodiment of an interface which shows a time estimate 38 generated for15 trial runs for selected parameters prior to starting theself-calibration process.

In a further embodiment, after an end-user chooses input parameters andoutput parameters, any known optimization algorithm could be employedduring the calibration process. A non-limiting example would be apowerful heuristic algorithm (e.g., genetic algorithm (GA), simulatedannealing, and downhill simplex) that requires an excessive andunpredictable number of trials, and thus may not be well-suited tocomputationally expensive simulations. Faster heuristics (e.g.,hill-climbing, gradient methods, and greedy algorithms) have been knownto produce poor solutions. By contrast, the embodiments of the subjectinvention can be specifically used to calibrate time-expensivesimulations. End-users can pro-actively customize trial values and runtimes for directed brute force (DBF) searching.

As mentioned above, the difference between brute force (BF) and“directed” brute force (DBF) based simulations are that DBF can uses arestricted set of trial values within a given database 60. In theearlier example Quick searching (3 trials) on one input parameter,Medium (5 trials) on a second, and Thorough (10 trials) on a third wouldlead to simulation that provides 3*5*10=150 possible solutions. Thenumber of trials for Quick, Medium, and Thorough can be flexibleadjustable in the embodiments of the subject invention. The database 60can offer intelligent defaults, but also allow customization.Embodiments of the subject invention are particularly useful with DBF.

In an alternative embodiment, default values for use with inputparameters could be determined on a sliding scale, such that a range ofvalues can be pre-selected for one or more input parameters. There are avariety of methods by which this can be accomplished. In one embodiment,there can be provided a selection device 65, such as, for example, aninput slider display, that allows a user to select from a large range ofvalues a smaller sub-range of values from which input values can beselected during calibration. FIG. 14 illustrates one example of inputsliders 65 being provided for those input parameters selected forcalibration. In a further embodiment, each constrained sub-range for aparticular input parameter can be set with default incrementalstep-sizes 67 to be used within the constrained range. For example, inFIG. 14, the input parameter entitled “Car Following Sensitivity” couldbe set to test values in the selected range at incremental steps of 0.1increments. Likewise, the input parameter entitled “MaximumNon-emergency Deceleration” could be set to test values in the selectedrange at incremental steps of 1.0. In one embodiment, the step-size 67for range selected with the input slider 65 can be pre-determined andset within the calibration software. In an alternative embodiment, thestep-size 67 for each selected range can also be selected by a user. Byway of non-limiting example, FIG. 14 illustrates a step-size dialog box70 in which a user can insert the stepsize to be used. A person withskill in the art would be able to determine any of a variety of methodsby which a range can be selectable by a user, as well as a step-size.Such variations are within the scope of this invention.

FIG. 6 illustrates one embodiment of a user-interface (UI) softwarescreen, for selection of inputs to be calibrated. This UI screen loadsthe input parameter database at runtime; thus calibrating a differentsimulation product can require switching databases. In one embodiment,an end-user can select, such as with a check mark, each input parameterto calibrate. In a further embodiment, the user can further select acalibration thoroughness level (e.g., Quick, Medium, or Thorough) foreach input selected.

When directed brute force (DBF) optimization is in effect, a total runtime estimate can be shown at the bottom of the screen. Advanceknowledge of computer run times can be advantageous. For example, if anend-user chose Quick searching (3 trials) on one input parameter, Mediumsearching (5 trials) on a second, and Thorough searching (10 trials) ona third, run time from an initial simulation could be multiplied by 150(i.e., 3*5*10), to produce a reasonable estimate. If a longer run timecan be tolerated, a more thorough searching on some parameters can beselected, or the number of parameters can be increased. If the estimatedrun time were uncomfortably high, the calibration levels can be adjustedor certain parameters can be removed from the optimization.

When Simultaneous Perturbation Stochastic Approximation (SPSA)optimization is in effect, run time estimates may not be possible unlessbased on the max allowed number of simulations. However, by allowing theend-user to easily and dynamically adjust range limits (continuous SPSA)or trial values (discrete SPSA) for any input parameter, SASCO's inputdata UI is expected to augment the efficiency of SPSA.

Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm hasgained favor as an algorithm for efficient optimization of complexsimulations. When an SPSA optimization algorithm is employed withembodiments of the subject invention, run time estimates may not bepossible unless based on the maximum allowed number of simulations.However, by allowing the end-user to easily and dynamically adjust rangelimits (continuous SPSA) or trial values (discrete SPSA) for any inputparameter, the embodiments of the subject invention can augment theefficiency of SPSA. For the discrete form of SPSA, the database 60 canbe made to specify explicit trial values for each input parameter 12.For the continuous fond′, the database 60 can adjust range limits foreach input parameter.

An alternative embodiment of the input parameter database would involvesoftware controls (e.g., sliders, checkboxes), to easily reduce thenumber of trial input values used during calibration. FIG. 39illustrates an example of slider controls currently used in the art todynamically adjust the range limits of inbound and outbound flighttimes, plus checkboxes to select price ranges. Similar controls could beused to dynamically adjust range limits of calibration input parameters,and/or specify a step-size between range limits.

FIG. 13 illustrates one embodiment of the operation of the calibrationmethod of the subject invention. As shown, when the run is launched, theselected optimization algorithm (e.g., DBF or SPSA) proceeds to minimizethe objective function, by testing acceptable input values from theinput data UI. Throughout the run the discrepancy between simulated andfield-measured results, a.k.a., the objective function, become lower andlower, thus indicating a better-calibrated model. In one embodiment, apost-run input parameter calibration interface screen 21 is providedthat shows the results of each trial 5 conducted by the self-calibrationprocess as it operates. FIG. 4 illustrates an example of this interfacescreen on which a file name 6 representing each candidate data set 40generated by each trial 5 run is listed, along with the % Difference 19,i.e., difference between simulation and empirical results, for eachcandidate data set 40. This allows an end-user to observe % Differencesas they are generated by the model for a particular candidate scenario.In the example shown in FIG. 4, the self-calibration method of thesubject invention has only generated results through trial 7 of the 15possible trials that it is scheduled to complete.

In a further embodiment, a user is provided with the option of abortinga self-calibration process during operation. Thus, if during observationof the self-calibration results, it becomes apparent that continuedexecution of the self-calibration process is unnecessary, a user canselect to abort the process before completion. This can save time andresources. In a particular embodiment, an interface screen can includeany of a variety of control features that a user can select to abort theself-calibration process. FIG. 4 illustrates an example of a post-runinput parameter calibration interface screen 21 with a stop button 39that can be used to abort a self-calibration process.

Once a self-calibration cycle has been completed, a user can observe thetrials 5 and select the candidate data set 40 providing the mostaccurate correlation to the ground truth values, such as empirical data,i.e., field-measured data. FIG. 2 illustrates an example of how thecalibration method of the subject invention can produce candidate datasets, which produce model-estimated results, which are sometimes closerto field-measured or empirical values. Input parameter values from thesecandidate data sets can then be imported to the simulation model, formore accurate modeling. More specifically, the candidate data sets bestcorrelated to the empirical data can be selected, and archived oruploaded, for use in future modeling.

In one embodiment, a user can be provided with the option of loadingparticular candidate trial values. FIG. 11 illustrates an example ofpost-run input parameter calibration interface screen overlaid with a“load calibration settings” dialog box 50, in which a user can choose toupload individual (candidate) trial values. With this example, if theuser selects “yes” the candidate values from the file named “34.trf”will be uploaded to the simulation model.

Once a simulation or trial has been conducted, simulated outputparameter values can be compared to field measured or empirical outputvalues, and a Difference value 19 can be generated for that particulartrial. In one embodiment, a Difference Value is displayed as apercentage of the difference between the simulated and empirical outputvalues. FIG. 3 illustrates one example where the Difference Value isrepresented as % Difference. FIGS. 4 and 6 illustrate a list of trials 5in the process of being conducted, for which those that are completed a% Difference has been calculated. In a further embodiment, variousstatistical analyses can be conducted on the Difference values 19 toprovide a user with information about overall results of theself-calibration. These analyses 45 can be provided to the user, forexample, as shown in FIG. 6. By way of example, the Difference values 19can be analyzed to determine which candidate data set provided theclosest fit to the measured output parameters, what is the mean value ofthe Difference for all of the trials, average standard deviation betweenthe trials, and other information that would be relevant to a user. Aperson with skill in the art would be able to determine any of numeroustypes of statistical analyses that can be conducted with the Differencevalues 19 or the underlying candidate data sets 40. Such variations arewithin the scope of the subject invention.

Prior to uploading to the simulation model a particular candidate dataset, it can be beneficial for a user to review the input parametervalues contained within the candidate data set. In one embodiment, userscan manually search the one or more individual data set(s) to reviewcandidate or calibrated parameter values therein. However, this canrequire a user to spend several minutes or longer selecting and openingindividual data sets. Another embodiment of the subject invention allowsa user to select and simultaneously view specific or even all of thecandidate data set values for a selected calibration run. This canprovide an end-user with more information about which input parametervalues produced the best fit between model-estimated and field-measuredor empirical observations. Additionally, this can allow a user toinspect various candidate values without necessarily importing them tothe simulation model; which could aid in their understanding of thecalibration process, as well as provide information about how to createfurther, more refined, self-calibration scenarios. FIG. 9 illustrates acandidate value interface screen 52 showing the actual values for aparticularly selected files called “80.trf” and a “load calibrationsettings” dialog box 50 thereon.

In one embodiment, a candidate value interface screen 52 can bedisplayed when a user selects a particular simulation or trial 40, suchas one shown in the example in FIG. 6 or FIG. 11. In a furtherembodiment, the “load calibration settings” dialog box 50 can bedisplayed on the candidate value interface screen 52, an example ofwhich is shown in FIG. 9. After reviewing the candidate data, the “loadcalibration” dialog box provides a user with the option of choosingwhether the candidate data should be uploaded or not. In a furtherembodiment, the self-calibration method can be repeated sequentiallyafter importing prior calibrated values, to continually refine accuracywhile minimizing run times.

Embodiments of the directed brute force calibration method describedherein can permit a simulation model to be calibrated to a desiredaccuracy. In a specific implementation, the calibration method can beused to customize a traffic simulation model for a particular area orregion or even a particular street. This can increase the effectivenessand usability of a model, while at the same time increasing the accuracyof the model for a given area.

Following are examples that illustrate procedures for practicing variousembodiments of the subject invention. These examples are provided forthe purpose of illustration only and should not be construed aslimiting. Thus, any and all variations that become evident as a resultof the teachings herein or from the following examples are contemplatedto be within the scope of the present invention.

Example 1: Method of Self-Calibration of the TSIS-CORSIM TrafficSimulation Model

TSIS-CORSIM is a microscopic traffic simulation software model forsignal systems, highway systems, freeway systems, or combinationsthereof. CORSIM (CORridor SIMulation) is an integrated set of two othermicroscopic traffic simulation models. The first, NETSIM, simulatestraffic on urban streets and the other, FRESIM, simulates traffic onhighways and freeways. TSIS (Traffic Software Integrated System) is adevelopment software program that enables a user to conduct trafficoperations analysis. TSIS allows a user to customize a set of tools,define and manage traffic analysis projects, define or create trafficnetworks and conduct traffic simulation analyses for interpretation. Thefollowing example is specifically directed to an implementation of thesubject method for calibration of the TSIS-CORSIM program.

Overview:

The TSIS-CORSIM self-calibration process is managed from TSIS Next,which is one of the available “front-end” interface applications forCORSIM simulation. The self-calibration feature also requires build 514(or higher) of the CORSIM simulation engine. Before initiatingself-calibration, a user should load a CORSIM data set (*.TRF) into TSISNext; and should perform at least one standard CORSIM run from withinTSIS Next, to generate the simulation outputs necessary for calibration.If a user a user wishes to calibrate time period-specific outputs, inaddition to, or instead of, cumulative outputs, a set of expanded CSVfiles should also be generated; which can be generated by turning on thecheckboxes under Options>Preferences>Output Files, prior to performing aCORSIM run.

The self-calibration features can be launched in TSIS Next by clickingon the Self-Calibration toolbar icon, or by selectingRun>Self-Calibration, or by right-clicking on a link in the Map View andselecting Self-Calibrate. The Output Parameters interface screen isshown in FIG. 3, and will be described first. The Input Parameters andRun Status interface screen will be illustrated and described later.

Simulated performance measure values, from the most recent run, areautomatically displayed for easy reference. Field-measured values can beentered for any number of output parameters provided on the list.Field-measured values may be entered for any temporal setting(cumulative and/or time period-specific), and for any spatial setting(global and/or link-specific). Priority weightings may be specified forany of the output parameters, if desired. Link-specific calibrationsettings can be accessed, or specified, by right-clicking on a link inthe Map View. Overall percent difference, between field-measured valuesand simulated values, is automatically displayed for easy reference.Finally, user-specified calibration settings are automatically archived(within <user_filename>.self).

The Input Parameters and Run Status interface screen is shown in FIG. 4,and will be described next.

A user can choose to calibrate any number of input parameters providedon the list. Freeway input parameters are not displayed for surface-onlynetworks, and vice-versa. Combined surface-and-freeway networks willpresent both FRESIM and NETSIM input parameters for selection. Theamount of searching (Quick, Medium, or Thorough) can be selected foreach input parameter. The resulting number of candidate “trial” valuesis then displayed next to each input parameter. A number of chosen inputparameters, together with the amount of searching for each parameter,combine to produce the estimated run time, displayed at the bottom ofthe interface screen. The estimated run time is a crucial element;because it allows a user a user to choose their own tolerable run time,prior to the run. When the number of chosen input parameters isincreased, or when the amount of searching is increased, the estimatedrun time increases simultaneously.

Therefore, a user can use judgment in choosing input parameters andsearch levels, which can improve their model accuracy within areasonable amount of time. For example, in surface-street networks thatdo not have permissive left turns in them, Permissive Left-TurnAcceptable Gaps (NETSIM) should not be chosen for self-calibration, asthis will blow up run times without improving model accuracy.

CORSIM input parameters for Mean Queue Discharge Headway (NETSIM) andCar Following Sensitivity Multiplier (FRESIM) are sometimes calibratedto contain unique values on certain links, and in certain areas of thetraffic network. Because of this, the pop-up dialog shown in FIG. 5 isprovided for these two parameters, to allow calibration on some linksbut not others.

Regarding the Run Status section of the interface screen, the Generatebutton is used to create trial data sets, prior to the self-calibrationrun.

Scenario: An end-user chooses Quick searching (3 trials) on one inputparameter, Medium searching (5 trials) on a second input parameter, andThorough searching (10 trials) on a third input parameter, which wouldgenerate 3*5*10=150 data sets, according to the directed brute forcemethodology.

The Start button is then used to initiate the self-calibration run,during which the percent difference for each trial run is displayed inreal-time. The Stop button can be used to abort the run prior tocompletion, if desired. The revised Input Parameters and Run Statusinterface screen is shown again, as illustrated in FIG. 6, but this timeat the conclusion of a run.

After the self-calibration run is completed, some of the trial runs mayexhibit percent differences lower than the original percent difference,thus indicating a better-calibrated traffic network. These trial datasets are then available for further use in the SelfCalibrate folder,which is a subfolder of the folder containing the original data set.Also following the self-calibration run, the mean value and standarddeviation are displayed. The standard deviation provides a measure ofmodel sensitivity to the input parameters that were just chosen andtested. Higher values of standard deviation indicate model results thatwere highly sensitive to the chosen input parameters.

Automatic Importing of Calibration Settings

Calibration settings can be automatically imported from specific trialdatasets, if desired. This can be done by clicking on the name of anytrial dataset, and a pop-up box, such as shown, for example in FIG. 7,will ask whether or not the calibration settings for that dataset shouldbe imported. This process is also demonstrated in the tutorialexercises.

Database of Input Parameters for Self-Calibration

The database of input parameters for self-calibration is shown in FIG.8.

By default, this database contains a few dozen preset input parameterswithin a few dozen text files. The default list of calibration inputparameters is shown below. This default list may change (sequence and/orcontent) in future software versions.

1. Entry Headway Seed

2. Traffic Stream Seed

3. Traffic Choice Seed

4. Vehicle Entry Headway

5. Maximum Network Initialization Time

6. Car Following Sensitivity Multiplier (FRESIM)

7. Car Following Sensitivity (FRESIM)

8. Time to Complete a Lane Change (FRESIM)

9. Minimum Entry Headway (FRESIM)

10. Percentage of Cooperative Drivers (FRESIM)

11. Lane Change Desire (FRESIM)

12. Lane Change Advantage (FRESIM)

13. Maximum Non-Emergency Deceleration (FRESIM)

14. Maximum Perceived Deceleration (FRESIM)

15. Time to Complete a Lane Change (NETSIM)

16. Deceleration Reaction Time (NETSIM)

17. Minimum Lane Changing Deceleration (NETSIM)

18. Mandatory Lane Change Deceleration (NETSIM)

19. Discretionary Lane Change Deceleration (NETSIM)

20. Lead Vehicle Deceleration (NETSIM)

21. Follower Vehicle Deceleration (NETSIM)

22. Driver Aggressiveness Factor (NETSIM)

23. Urgency Threshold (NETSIM)

24. Safety Factor (NETSIM)

25. Percentage of Cooperative Drivers (NETSIM)

26. Minimum Lane Change Headway (NETSIM)

27. Maximum Lane Change Headway (NETSIM)

28. Lane Change Distance (NETSIM)

29. Queue Discharge and Lost Time Distribution (NETSIM)

30. Mean Queue Discharge Headway (NETSIM)

31. On-Ramp Speed for Upstream Lane Changes (FRESIM)

32. Left-Turn Jumpers (NETSIM)

33. Turning Speed (NETSIM)

34. Probability of Joining Queue Spillback (NETSIM)

35. Probability of Left-Turn Lagger (NETSIM)

36. Near-Side Gap Acceptance at a Sign (NETSIM)

37. Far-Side Gap Acceptance at a Sign (NETSIM)

38. Amber Interval Response (NETSIM)

39. Permissive Left-Turn Acceptable Gaps (NETSIM)

40. Permissive Right-Turn Acceptable Gaps (NETSIM)

41. Pedestrian Delays during Weak Interaction (NETSIM)

42. Pedestrian Delays during Strong Interaction (NETSIM)

43. Free Flow Speed Distribution (NETSIM)

44. Free Flow Speed Distribution (FRESIM)

45. Distribution of Lane Change Distance (NETSIM)

46. Driver Familiarity with Paths (NETSIM)

47. On-Ramp Anticipatory Lane Change Distance (FRESIM)

48. Off-Ramp Reaction Distance (FRESIM)

49. Mean Start-Up Lost Time (NETSIM)

The input parameter database can be customized by a user if desired.These text files can be created or edited quickly, usually withinminutes, and are automatically loaded by the program at run time. Bychanging the numeric file names, a user can change the order of inputparameters displayed on-screen. By deleting certain text files in thedatabase, a user can remove input parameters from the list. By changingthe numbers inside the text files, a user can re-define the amount ofsearching associated with Quick-Medium-Thorough, or change the existingtrial values themselves. To do this, a user would often need to refer tothe CORSIM Reference Manual, to access the definition of CORSIM dataformats. Adding text files to the list could add to the list of inputparameters; but the current program is designed to handle either datatypes in the preset database, or data types already present in theoriginal data set. For example, if a user wishes to calibrate data type150 for HOT lanes, then data type 150 must be present in the originaldata set, because data type 150 is not included in the default databaseof input parameters.

Data types (a.k.a. record types) available in the default database areas follows: 2, 11, 20, 68, 70, 81, 140, 141, 142, 143, 144, 145, 146,147, 152, 153.

A user can exercise judgment with regard to the random number seeds(vehicle entry headway, traffic choice, traffic flow), which sit atopthe default list of input parameters. If these seed parameters are notincluded in a self-calibration run, it is possible that some of theindividual trial results may be “outliers”. For better statisticalcertainty, it can be helpful to perform random number seed “calibration”at the appropriate times, to observe a fuller range of possibleoutcomes. This recommended procedure is illustrated in the TutorialExamples section.

The calibration software allows users to observe all candidate datavalues at a glance, for any trial. Typically, a user can manually searchone or more datasets, to review the candidate or calibrated parametervalues provided within each dataset. FIG. 11 shows an interface screenon which candidate values for the three parameters in the 80.trf trialdata set are displayed. The calibration settings dialog box shown inFIG. 11 allows a user to decide whether to upload the candidate values.By selecting “yes” in the pop-up dialog box, the calibration method willcontinue using the pre-set candidate values. If “no” is selected, a userwill have the option of editing the candidate values.

Example 2: Tutorial for Calibration of Surface Street Simulation

The first tutorial example is a simple surface-street section with twosignalized intersections. The intersections are separated by 1800 feet.The volume, timing, and laneage details were derived from a 2010 HighwayCapacity Manual example problem, in which access points between theintersections were deleted. The simple link-node geometry is shown inFIG. 10.

Before beginning the normal self-calibration runs, it would be advisableto perform some preliminary random number seed analysis, to determinewhether the simulated performance measures actually representnon-typical results. However, field-measured performance measure valuesshould be entered prior to any type of analysis. The steps to set up apreliminary random number seed analysis are as follows:

-   -   Launch ISIS Next.    -   Open Sample #1.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Next, field-measured values are entered in the Output Parametersinterface screen. The values to be used for this exercise are listedbelow. These values can be entered by clicking on the “Self-Calibration”toolbar icon, and then switching between links using the combo box; orby right-clicking on specific links in the Map View, and then selectingthe “Self-Calibrate” menu item. In each case, after entering the numericvalue, it is then necessary to turn on the checkbox under the“Self-Calibrate?” column.

SpeedAverage values for Cumulative Link-Specific Surface (NETSIM)

-   -   Enter 22.96 for links 1--->2 and 2--->1        Delay Control per Vehicle values for Cumulative Link-Specific        Surface (NETSIM)    -   Enter 19.30 for links 301--->1 and 402--->2    -   Enter 25.10 for links 2--->1 and 1--->2    -   Enter 39.40 for links 101--->1, 201--->1, 102--->2, and 202--->2

After entering in all of these values, the “total percent difference”should be displayed as 25.1%. Now, the preliminary random number seedanalysis can begin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data row that says Entry Headway Seed, turn on the        checkbox under the Self-Calibrate? column    -   At the far left side of this same data row, select Thorough        under the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 32.1% is displayed, which issignificantly different than the original percent difference (25.1%).When this process is repeated with only the Traffic Stream Seedselected, a Mean Value of 29.2% is displayed, which is alsosignificantly different than the original. When this process is repeatedwith only the Traffic Choice Seed selected, a Mean Value of 27.2% isdisplayed, which is perhaps not significantly different than theoriginal. Finally, Entry Headway Seed and Traffic Stream Seed aresimultaneously selected for Thorough searching (100 total runs, timeestimate 2 minutes 17 seconds).

FIG. 7 below shows that data file 34.trf contains a % Difference valuethat is closest to the Mean Value for all 100 trial datasets. Therefore,it might be safer to perform calibration using random number seedswithin 34.trf, instead of random number seeds within the originaldataset.

Random number seeds from 34.trf can be automatically imported byclicking on the table cell that says “34.trf”. After this, performancemeasures based on those random number seeds can be automaticallygenerated by performing another CORSIM run, without even leaving theSelf Calibration interface screen. Just click “Yes” when the programsays “Refresh output by performing a CORSIM run?”, as shown in FIG. 11.

Now that the random number seeds have been checked, the self-calibrationprocess can continue. Because NETSIM has a large number of calibrationparameters available for selection, some judgment by a user is neededfor choosing a smaller set of parameters for self-calibration. This isbecause simultaneous calibration of all parameters would result in anunacceptable computer run time. Certain input parameters should beomitted from self-calibration on the basis of traffic networkcharacteristics. For networks with no permissive left-turn movements,the associated input parameters (i.e., Permissive Left-Turn AcceptableGaps, Probability of Left-Turn Lagger, Left-Turn Jumpers) should not bechosen for self-calibration. The same holds for networks with no queuespillback (Probability of Joining Spillback), networks with noright-turns-on-red (Permissive Right-Turn Acceptable Gaps), or networkswith no sign-controlled intersections (Near-Side Gap Acceptance at aSign, Far-Side Gap Acceptance at a Sign).

In future versions of the software, it is hoped that research studiesfor sensitivity analysis will help to prioritize the input parameters interms of importance, such that the parameters tending to affect resultsthe most would appear near the top of the list by default. For now, aset of preliminary self-calibration runs was performed to gauge theeffect of each parameter. The preliminary runs were performed asfollows:

-   -   Click on the tab that says Input Parameters and Run Status    -   Choose Vehicle Entry Headway as the only parameter chosen for        self-calibration    -   Set the Searching level to Quick    -   Click on the Generate button    -   Click on the Start button    -   At the end of the run, if the Best Value is lower than the        Original % Difference value by more than 0.1%, make a note of        the Standard Deviation value

This run showed that the trial Vehicle Entry Headways did not produce alower Best Value. Therefore, the Standard Deviation value (0.6) is notrecorded for Vehicle Entry Headway. After a few minutes of preliminaryruns in this manner, there were sixteen input parameters that showedpromise:

 1. Maximum Network Initialization Time 1.4  2. Minimum Lane ChangingDeceleration (NETSIM) 1.5  3. Discretionary Lane Change Deceleration(NETSIM) 1.4  4. Lead Vehicle Deceleration (NETSIM) 3.1  5. FollowerVehicle Deceleration (NETSIM) 0.9  6. Driver Aggressiveness Factor(NETSIM) 0.7  7. Safety Factor (NETSIM) 0.7  8. Percentage ofCooperative Drivers (NETSIM) 0.4  9. Lane Change Distance (NETSIM) 0.910. Mean Queue Discharge Headway (NETSIM)* 2.5 11. Left-Turn Jumpers(NETSIM) 3.0 12. Turning Speed (NETSIM) 1.4 13. Amber Interval Response(NETSIM) 2.2 14. Free Flow Speed Distribution (NETSIM) 2.9 15. DriverFamiliarity with Paths (NETSIM) 1.6 16. Mean Start-Up Lost Time (NETSIM)1.1 *All links selected

Even at the Quick searching level, simultaneous calibration of allsixteen parameters produces a run time estimate that is consideredunacceptably high. Therefore, these output parameters wereself-calibrated individually at the Thorough level, and the bestsolution was automatically loaded after each run. After a few minutes ofsequential self-calibration runs in this manner, the percent differenceimprovements (original percent difference was 31.9%) were as follows:

 1. Maximum Network Initialization Time 28.5  2. Minimum Lane ChangingDeceleration (NETSIM) 28.5  3. Discretionary Lane Change Deceleration(NETSIM) 28.5  4. Lead Vehicle Deceleration (NETSIM) 28.5  5. FollowerVehicle Deceleration (NETSIM) 26.5  6. Driver Aggressiveness Factor(NETSIM) 25.7  7. Safety Factor (NETSIM) 25.7  8. Percentage ofCooperative Drivers (NETSIM) 25.7  9. Lane Change Distance (NETSIM) 25.710. Mean Queue Discharge Headway (NETSIM)* 21.9 11. Left-Turn Jumpers(NETSIM) 19.1 12. Turning Speed (NETSIM) 19.1 13. Amber IntervalResponse (NETSIM) 18.7 14. Free Flow Speed Distribution (NETSIM) 16.015. Driver Familiarity with Paths (NETSIM) 15.8 16. Mean Start-Up LostTime (NETSIM) 15.8 *All links selected

At this stage, it may appear that the calibration exercise has doubled(31.9% percent difference, reduced to 15.8%) the accuracy of the model.However, it is still possible that the 15.8% result represents anon-typical result, due to stochastic variation. Thus, it is advisableto perform some “post-calibration” random number seed analysis. Bychoosing Medium searching for all three random number seeds, it ispossible to set up a randomness analysis run with only 125 simulations(run time estimate 2 minutes 40 seconds). After this post-calibrationanalysis run, the Mean Value is displayed as 20.8%, with 33.trf as theMean Dataset. Clicking on 33.trf automatically loads these random numberseeds, and File>Save As can now be used to save this calibrated network.

At this point, a user might want to review the calibrated network(animation, inputs, outputs) for possible problems and anomalies. Inaddition, manual (non-automated) “fine-tuning” calibration couldoptionally be performed at this stage.

In summary, at the outset, it appeared that the percentage difference(between simulated and field-measured performance) was 25.1%, butrandomness analysis revealed the actual difference was closer to 31.9%.In less than an hour, it was possible to reduce the percentagedifference from 32% to around 21%. Since the calibration of individualsettings was done sequentially, instead of simultaneously, it ispossible that the best combination of settings was not yet located. Thesequential process was used to keep run times low, but a simultaneousprocess (with a small number of input parameters) might be preferable.To reduce the percentage difference further below 21%, a user might wantto re-consider the overall list of options below.

1. Correction of fundamental input data (e.g., volume, timing, laneage)errors 2. Reconciliation of inconsistent performance measuredefinitions* 3. Collection of more accurate field-measured values 4.Elimination of simulation software limitations* 5. Correction ofsimulation software bugs* 6. Manual (non-automated) calibration 7.Automated self-calibration *usually handled by the simulation softwaredevelopers

In this exercise, all calibration input parameters were treated as “fairgame”, such that changing those parameters would be acceptable to theclient. In a real-world project, a user could use judgment to avoidself-calibration of 1) parameters the client does not want changed, and2) parameters that are not applicable (e.g., Probability of JoiningSpillback in an undersaturated network) to the given network.Self-calibration of non-applicable parameters would not interfere withaccurate model results; but it would waste time, because non-applicableparameters have no impact on model results.

Example 3: Tutorial for Calibration of Multi-Period Freeway Simulation

This tutorial example is an extended freeway facility with elevensegments, and five time periods. Volume and laneage details were derivedfrom a 2010 Highway Capacity Manual example problem. The link-nodegeometry is shown in FIG. 10.

Before beginning the normal self-calibration runs, it would be advisableto perform some preliminary random number seed analysis, to determinewhether the simulated performance measures actually representnon-typical results. However, field-measured performance measure valuesshould be entered prior to any type of analysis. The steps to set up apreliminary random number seed analysis are as follows:

-   -   Launch TSIS Next.    -   Open Sample #2.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Next, field-measured values are entered in the Output Parametersinterface screen. The values to be used for this exercise are listedbelow. These values can be entered by clicking on the “Self-Calibration”toolbar icon, and then switching between links using the combo box; orby right-clicking on specific links in the Map View, and then selectingthe “Self-Calibrate” menu item. In each case, after entering the numericvalue, it is then necessary to turn on the checkbox under the“Self-Calibrate?” column.

DensityPerLane values for Time Period 3 Link-Specific Freeway (FRESIM)

-   -   Enter 29.30 for link 101--->201    -   Enter 34.20 for link 301--->402    -   Enter 31.90 for link 501--->602    -   Enter 35.80 for link 701--->801    -   Enter 37.70 for link 1101--->1102        SpeedAverage values for Time Period 3 Link-Specific Freeway        (FRESIM)    -   Enter 59.40 for link 101--->201    -   Enter 57.10 for link 301--->402    -   Enter 58.30 for link 501--->602    -   Enter 56.10 for link 701--->801    -   Enter 55.00 for link 1101--->1102

After entering in all of these values, the “total percent difference”should be displayed as 3.3%. Now, the preliminary random number seedanalysis can begin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data row that says Entry Headway Seed, turn on the        checkbox under the Self-Calibrate? column    -   At the far left side of this same data row, select Medium under        the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 3.3% is displayed, which isidentical to the original percent difference. When this process isrepeated with only the Traffic Stream Seed selected, a Mean Value of3.3% is again displayed. When this process is repeated with only theTraffic Choice Seed selected, a Mean Value of 3.3% is again displayed.Although the Entry Headway and Traffic Choice seeds produced somevariability in results, the overall Mean Value was still 3.3%. Thismeans the original dataset probably contains random number seeds thatproduce typical results, and it is probably safe to performself-calibration without changing the original random number seeds.

Now that the random number seeds have been checked, the self-calibrationprocess can continue. Because FRESIM has a large number of calibrationparameters available for selection, some user judgment is needed forchoosing a smaller set of parameters for self-calibration. This isbecause simultaneous calibration of all parameters would result in anunacceptable computer run time.

Analysis of sensitivity analysis can help to prioritize the inputparameters in terms of importance, such that the parameters tending toaffect results the most would appear near the top of the list bydefault. For now, a set of preliminary self-calibration runs wasperformed to gauge the effect of each parameter. The preliminary runswere performed as follows:

-   -   Click on the tab that says Input Parameters and Run Status    -   Choose Vehicle Entry Headway as the only parameter chosen for        self-calibration    -   Set the Searching level to Thorough    -   Click on the Generate button    -   Click on the Start button    -   At the end of the run, if the Best Value is lower than the        Original % Difference value by more than 0.1%, then make a note        of the Best Value

This run showed that the trial Vehicle Entry Headways produced a BestValue of 2.8%. Therefore, the Best Value (2.8) is recorded for VehicleEntry Headway. After a few minutes of preliminary runs in this manner,there were fourteen input parameters that showed promise:

 1. Vehicle Entry Headway 2.8  2. Maximum Network Initialization Time2.7  3. Car Following Sensitivity Multiplier (FRESIM)* 2.8  4. CarFollowing Sensitivity (FRESIM) 2.4  5. Time to Complete a Lane Change(FRESIM) 3.1  6. Minimum Entry Headway (FRESIM) 3.0  7. Lane ChangeDesire (FRESIM) 2.8  8. Lane Change Advantage (FRESIM) 2.7  9. MaximumNon-Emergency Deceleration (FRESIM) 2.4 10. Maximum PerceivedDeceleration (FRESIM) 3.1 11. On-Ramp Speed for Upstream Lane Changes(FRESIM) 2.9 12. Free Flow Speed Distribution (FRESIM) 2.5 13. On-RampAnticipatory Lane Change Distance (FRESIM) 2.9 14. Off-Ramp ReactionDistance (FRESIM) 2.8 *All links selected

For this exercise, suppose that user judgment was used to eliminate CarFollowing Sensitivity Multiplier and Maximum Network Initialization Timefrom consideration. Car Following Sensitivity Multiplier and CarFollowing Sensitivity (global) are somewhat redundant, and Car FollowingSensitivity Multiplier is primarily used to affect operations onspecific individual links. Maximum Network Initialization Time produceda low Best Value in its first couple of trials, but this was because thenetwork failed to reach equilibrium.

Even at the Quick searching level, simultaneous calibration of alltwelve remaining parameters produces a run time estimate that isconsidered unacceptably high. Therefore, these output parameters wereself-calibrated individually at the Thorough level, and the bestsolution was automatically loaded after each run. After a few minutes ofsequential self-calibration runs in this manner, the percent differenceimprovements (original percent difference was 3.3%) were as follows:

 1. Vehicle Entry Headway 2.8  2. Car Following Sensitivity (FRESIM) 2.1 3. Time to Complete a Lane Change (FRESIM) 2.1  4. Minimum EntryHeadway (FRESIM) 1.9  5. Lane Change Desire (FRESIM) 1.9  6. Lane ChangeAdvantage (FRESIM) 1.9  7. Maximum Non-Emergency Deceleration (FRESIM)1.9  8. Maximum Perceived Deceleration (FRESIM) 1.9  9. On-Ramp Speedfor Upstream Lane Changes (FRESIM) 1.9 10. Free Flow Speed Distribution(FRESIM) 1.9 11. On-Ramp Anticipatory Lane Change Distance (FRESIM) 1.912. Off-Ramp Reaction Distance (FRESIM) 1.9

At this stage, it may appear that the calibration exercise has nearlydoubled (3.3% percent difference, reduced to 1.9%) the accuracy of themodel. However, it is still possible that the 1.9% result represents anon-typical result, due to stochastic variation. Thus, it is advisableto perform some “post-calibration” random number seed analysis.Repeating the same steps from the preliminary random number seedanalysis, only the Traffic Choice Seed appears to have any impact onresults. After this post-calibration analysis run, the Mean Value isdisplayed as 2.4%, with 1.trf as the Mean Dataset. Clicking on 1.trfautomatically loads these random number seeds, and File>Save As can nowbe used to save this calibrated network with another name.

At this point, a user might want to review the calibrated network(animation, inputs, outputs) for possible problems and anomalies. Inaddition, manual (non-automated) “fine-tuning” calibration couldoptionally be performed at this stage.

To summarize, it appeared at the beginning that the percentagedifference (between simulated and field-measured performance) was 3.3%,and random number analysis confirmed this to be a typical result forvarious seeds. In a short period of time, it was possible to reduce thepercentage difference from 3.3% to 2.4%. Since the calibration ofindividual settings was done sequentially, instead of simultaneously, itis possible that the best combination of settings was not yet located.The sequential process was used to keep run times low, but asimultaneous process (with a small number of input parameters) might bepreferable. To reduce the percentage difference further below 2.4%, auser might want to re-consider the overall list of options below.

1. Correction of fundamental input data (e.g., volume, timing, laneage)errors 2. Reconciliation of inconsistent performance measuredefinitions* 3. Collection of more accurate field-measured values 4.Elimination of simulation software limitations* 5. Correction ofsimulation software bugs* 6. Manual (non-automated) calibration 7.Automated self-calibration *usually handled by the simulation softwaredevelopers

In this exercise, all calibration input parameters were treated as “fairgame”, such that changing those parameters would be acceptable to theclient. In a real-world project, a user would use judgment to avoidself-calibration of 1) parameters the client does not want changed, and2) parameters that are not applicable to the given network.Self-calibration of non-applicable parameters would not interfere withaccurate model results; but it would waste time, because non-applicableparameters have no impact on model results.

Example 4: User's Guide to Self-Calibration of TSIS-CORSIM SimulationProgram

This user's guide describes the automated self-calibration features, formicro-simulation of traffic operations, within the TSIS-CORSIM softwarepackage.

I. How to Use SASCO in TSIS-CORSIM

The TSIS-CORSIM SASCO process is managed from TSIS Next, which is one ofthe available “front-end” interface applications for CORSIM simulation.The SASCO features also require build 514 (or higher) of the CORSIMsimulation engine. Before initiating SASCO, the user must load a CORSIMdata set (*.TRF) into TSIS Next; and must perform at least one standardCORSIM run from within TSIS Next, to generate the simulation outputsnecessary for calibration. If the engineer wishes to calibrate timeperiod-specific outputs, in addition to or instead of cumulativeoutputs, it is also necessary to generate a set of expanded CSV files;which can be generated by turning on the checkboxes underOptions>Preferences>Output Files, prior to performing a CORSIM run.

The SASCO features can be launched in TSIS Next by clicking on theSelf-Calibration toolbar icon, or by selecting Run>Self-Calibration, orby right-clicking on a link in the Map View and selectingSelf-Calibrate. The Output Parameters screen is illustrated in FIG. 19,and will be described first. The Input Parameters and Run Status screenwill be illustrated and described later.

Simulated performance measure values, from the most recent run, areautomatically displayed for easy reference. Field-measured values can beentered for any number of output parameters provided on the list.Field-measured values may be entered for any temporal setting(cumulative and/or time period-specific), and for any spatial setting(global and/or link-specific). For links with at least one surveillancedetector, output parameters generated by these detectors are availablefor self-calibration, near the bottom of the (time period-specific)output parameter list. For intersections under actuated control,phase-specific green times are available for self-calibration, near thebottom of the Global (time period-specific) output parameter list.Priority weightings may be specified for any of the output parameters,if desired. Link-specific calibration settings can be accessed, orspecified, by right-clicking on a link in the Map View. Node-specificand phase-specific settings are also available, primarily forsensitivity analysis and optimization; this functionality is explainedlater in more detail. Overall percent difference, between field-measuredvalues and simulated values, is automatically displayed for easyreference. Finally, user-specified calibration settings areautomatically archived (within <user_filename>.self).

The Input Parameters and Run Status screen is illustrated in FIG. 20,and will be described next.

The user can choose to calibrate, optimize, or analyze any number ofinput parameters provided on the list. Freeway input parameters are notdisplayed for surface-only networks, and vice-versa. Combinedsurface-and-freeway networks will present both FRESIM and NETSIM inputparameters for selection. The amount of searching (Quick, Medium, orThorough) can be selected for each input parameter. The resulting numberof candidate “trial” values is then displayed next to each inputparameter. The number of chosen input parameters, together with theamount of searching for each parameter, are combined to produce theestimated run time, displayed at the bottom of the screen. The estimatedrun time is a crucial element; because it allows the engineer to choosetheir own tolerable run time, prior to the run. When the number ofchosen input parameters is increased, or when the amount of searching isincreased, the estimated run time increases simultaneously.

Therefore, the engineer must use judgment in choosing input parametersand search levels, which can improve their model accuracy within areasonable amount of time. For example, in surface-street networks thatdo not have permissive left turns in them, Permissive Left-TurnAcceptable Gaps (NETSIM) should not be chosen for self-calibration, asthis will blow up run times without improving model accuracy.

CORSIM input parameters are sometimes calibrated to contain uniquevalues on certain links, and in certain areas of the traffic network.The list seen in FIG. 21 shows a certain number of input parameter namesthat begin with the phrase “Link-Specific”. These same Link-Specificinput parameters are also available in “global” form, in other areas ofthe list, when needed. The pop-up dialog box in FIG. 22 is provided forthese Link-Specific parameters, to allow automated calibration on somelinks but not others. When all link numbers have a check mark next tothem, Link-Specific parameters produce the same results as their“global” counterparts.

Regarding the Run Status section of the screen, the Generate button isused to create trial data sets, prior to the self-calibration run.

Example

If the user chooses Quick searching (3 trials) on one input parameter,Medium searching (5 trials) on a second input parameter, and Thoroughsearching (10 trials) on a third input parameter, this would generate3*5*10=150 data sets, according to the directed brute force methodology.

The Start button is then used to initiate the self-calibration run,during which the percent difference for each trial run is displayed inreal-time. The Stop button can be used to abort the run prior tocompletion, if desired.

The Input Parameters and Run Status screen is illustrated again in FIG.23, but this time at the conclusion of a run. After the self-calibrationrun is completed, some of the trial runs may exhibit percent differenceslower than the original percent difference, thus indicating abetter-calibrated traffic network. These trial data sets are thenavailable for further use in the SelfCalibrate folder, which is asubfolder of the folder containing the original data set. Also followingthe self-calibration run, the mean value and standard deviation aredisplayed. The standard deviation provides a measure of modelsensitivity to the input parameters that were just chosen and tested.Higher values of standard deviation indicate model results that werehighly sensitive to the chosen input parameters.

II. Automatic Viewing and Importing of Calibration Settings

Calibration settings can be automatically viewed and/or imported fromspecific trial datasets, if desired. Simply click on the name of anytrial dataset shown on the left side of the screen, shown in FIG. 24 anda pop-up box will ask whether or not the calibration settings for thatdataset should be imported, while simultaneously listing values thatwould be imported. This process is also demonstrated in the tutorialexercises. Calibration settings can be viewed before, during, or afterany self-calibration run. Calibration settings can be imported before orafter any self-calibration run, but not during the run.

III. Database of Input Parameters

The database of input parameters is illustrated in FIG. 25. By default,this database contains a few dozen preset input parameters within a fewdozen text files. The default list of input parameters is shown below.This default list may change (sequence and/or content) in futuresoftware versions.

 1. Entry Headway Seed  2. Traffic Stream Seed  3. Traffic Choice Seed 4. Vehicle Entry Headway  5. Maximum Network Initialization Time  6.Car Following Sensitivity Multiplier (FRESIM)  7. Car FollowingSensitivity (FRESIM)  8. Time to Complete a Lane Change (FRESIM)  9.Minimum Entry Headway (FRESIM) 10. Percentage of Cooperative Drivers(FRESIM) 11. Lane Change Desire (FRESIM) 12. Lane Change Advantage(FRESIM) 13. Maximum Non-Emergency Deceleration (FRESIM) 14. MaximumPerceived Deceleration (FRESIM) 15. Time to Complete a Lane Change(NETSIM) 16. Deceleration Reaction Time (NETSIM) 17. Minimum LaneChanging Deceleration (NETSIM) 18. Mandatory Lane Change Deceleration(NETSIM) 19. Discretionary Lane Change Deceleration (NETSIM) 20. LeadVehicle Deceleration (NETSIM) 21. Follower Vehicle Deceleration (NETSIM)22. Driver Aggressiveness Factor (NETSIM) 23. Urgency Threshold (NETSIM)24. Safety Factor (NETSIM) 25. Percentage of Cooperative Drivers(NETSIM) 26. Minimum Lane Change Headway (NETSIM) 27. Maximum LaneChange Headway (NETSIM) 28. Lane Change Distance (NETSIM) 29. QueueDischarge and Lost Time Distribution (NETSIM) 30. Mean Queue DischargeHeadway (NETSIM) 31. On-Ramp Anticipatory Lane Change Speed (FRESIM) 32.Left-Turn Jumpers (NETSIM) 33. Turning Speed (NETSIM) 34. Probability ofJoining Queue Spillback (NETSIM) 35. Probability of Left-Turn Lagger(NETSIM) 36. Near-Side Gap Acceptance at a Sign (NETSIM) 37. Far-SideGap Acceptance at a Sign (NETSIM) 38. Amber Interval Response (NETSIM)39. Permissive Left-Turn Acceptable Gaps (NETSIM) 40. PermissiveRight-Turn Acceptable Gaps (NETSIM) 41. Pedestrian Delays during WeakInteraction (NETSIM) 42. Pedestrian Delays during Strong Interaction(NETSIM) 43. Free Flow Speed Distribution (NETSIM) 44. Free Flow SpeedDistribution (FRESIM) 45. Distribution of Lane Change Distance (NETSIM)46. Driver Familiarity with Paths (NETSIM) 47. On-Ramp Anticipatory LaneChange Distance (FRESIM) 48. Off-Ramp Reaction Distance (FRESIM) 49.Mean Start-Up Lost Time (NETSIM) 50. Desired Free-Flow Speed (NETSIM)*51. Desired Mainline Free-Flow Speed (FRESIM)* 52. Desired RampFree-Flow Speed (FRESIM)* 53. Entry Node Volume* 54. Entry Node PercentTrucks* 55. Entry Node Percent Carpools* 56. Entry Node Percent HOVViolators* 57. Link-Specific Car Following Multiplier (FRESIM) 58.Link-Specific Mean Discharge Headway (NETSIM) 59. Link-Specific On-RampLane Change Speed (FRESIM) 60. Link-Specific On-Ramp Lane ChangeDistance (FRESIM) 61. Link-Specific Off-Ramp Reaction Distance (FRESIM)62. Link-Specific Start-Up Lost Time (NETSIM) 63. Link-Specific DesiredFree-Flow Speed (NETSIM)* 64. Link-Specific Desired Free-Flow Speed(FRESIM)* 65. Surface Node Yield Point (NETSIM)** 66. Freeway NodeMetering Headway (FRESIM)** *provided mainly for the purpose ofsensitivity analysis **provided mainly for optimization

The input parameter database can be customized by the user if desired.These text files can be created or edited within minutes, and areautomatically loaded by the program at run time. By changing the numericfile names, the user can change the order of input parameters displayedon-screen. By deleting certain text files in the database, the user canremove input parameters from the list. By changing the numbers insidethe text files, the user can re-define the amount of searchingassociated with Quick-Medium-Thorough, or change the existing trialvalues themselves. To do this, the user would often need to refer to theCORSIM Reference Manual, to access the definition of CORSIM dataformats. Adding text files to the list could add to the list of inputparameters; but the current program is designed to handle either datatypes in the preset database, or data types already present in theoriginal data set. For example, if the user wishes to calibrate datatype 150 for HOT lanes, then data type 150 must be present in theoriginal data set, because data type 150 is not included in the defaultdatabase of input parameters.

Data types (a.k.a. record types) available in the default database arelisted below:

2, 11, 20, 50, 68, 70, 81, 140, 141, 142, 143, 144, 145, 146, 147, 152,153

For example, if the engineer wanted to analyze the impact of left-turnpocket lengths, they could add a new text file containing trial valuesfor left-turn pocket length into the input database. Analyzing theimpact of left-turn pocket lengths would usually be consideredsensitivity analysis as opposed to calibration; sensitivity analysis isdiscussed in more detail in the upcoming section. Left-turn pocketlength is not included in the default database of input parametersbecause 1) they're only applicable to intersection approach links, andgenerate fatal errors when coded for other links, 2) their value can'texceed the length of the link that they're on, and 3) their absence isuseful in providing users guide examples for database customization(this section), and sensitivity analysis (next section).

It can be assumed that the default database of input parameters(typically available under C:\Program Files (x86)\FHWA\TSIS6.3\Database)contains 66 text files, such that left-turn pocket length will be textfile #67. The CORSIM Reference Manual specifies that left-turn pocketlengths are stored on record type 11, columns 13-16. It is importantthat this be a link-specific input parameter, so that only intersectionapproach links can be included in the analysis. It is also important toprovide trial values that will not exceed the link length. Samplecontents of “67.txt” are shown in FIG. 26.

The engineer must also exercise judgment with regard to the randomnumber seeds (vehicle entry headway, traffic choice, traffic flow),which sit atop the default list of input parameters. If these seedparameters are not included in a self-calibration run, there's a riskthat some of the individual trial results may be “outliers”. For betterstatistical certainty, it might help to perform random number seed“calibration” at the appropriate times, to observe a fuller range ofpossible outcomes. This recommended procedure is illustrated in theTutorial Examples section.

IV. Sensitivity Analysis

The SASCO features can also be used to facilitate sensitivity analysis(SA), and this process is demonstrated below in Tutorial Example #4.Practitioners and researchers have long relied on SA to gain a betterunderstanding of the modeling process, and to identify which inputparameters have the strongest impact on model results. Input parametersnot normally associated with calibration (e.g., desired free-flow speed,entry node volume, percent trucks, etc.) can now be examined quickly, inan automated manner, to determine their specific impact on results.However, the input parameters most commonly associated with calibrationcan be quickly analyzed via automated SA as well.

Regarding output parameters, engineers can still select their desiredperformance measures on the Output Parameters screen, but for SA itmight be easier in many cases to enter Measured values that actuallymatch the Simulated values. This way, the overall percent difference(between field-measured values and simulated values) will be displayedas 0.0 for existing conditions, prior to SA. Because the primary purposeof SA is to understand the impact of certain input parameters onsimulation model results, field-measured performance becomes a lowerpriority in the context of SA.

The previous section (Database of Input Parameters) provided an exampleon how to customize the input parameter database. This database can becustomized by modifying or deleting existing input parameters; but inthis example a brand-new input parameter (link-specific left-turn pocketlength) was added, in the form of “67.txt”. Once this text file has beenadded to the input database, link-specific left-turn pocket length willbe displayed at the bottom of the Input Parameters and Run Status screenat run-time. After selecting this new input parameter from the availablelist, SA is performed by clicking on “Generate”, followed by “Start”.Once this series of simulation runs is completed, SA results can beviewed in a tabular format; by switching over to the Output Parametersscreen, and then clicking on the “S.A.” button. Note that this buttononly becomes visible after trial datasets have been generated on theInput Parameters screen. The SA feature is powerful because it canautomatically contrast virtually any set of simulation inputs andoutputs, globally or link-specific, and cumulatively or TP-specific. Anexample of the SA results, for link-specific left-turn pocket length, isillustrated in FIG. 27.

In an earlier section it was mentioned that a pop-up dialog is providedfor Link-Specific input parameters, to allow automated calibration onsome links but not others. For the entry node input parameters (volume,percent trucks, percent carpools, and percent HOV violators) typicallyassociated with SA, it is also true that a pop-up dialog is provided forEntry Node-Specific parameters, to allow automated SA on selected entrynodes but not others.

The Federal Highway Administration Office of Operations Research andDevelopment has released a report (FHWA-HRT-04-131) for which extensiveSA (45,000 runs) was performed for CORSIM, considering a wide variety oftraffic networks. The findings of this SA research may be of use in thecontext of CORSIM self-calibration, because engineers might choose tofocus on calibrating those input parameters having the strongest impacton results.

V. Optimization

The SASCO features can also be used to facilitate optimization, and thisprocess is demonstrated below in Tutorial Example #5. Practitioners andresearchers have long focused on optimization of cycle length, phasingsequence, green splits, and offsets. Input parameters not normallyoptimized (e.g., detector lengths, ramp meter timings, esoteric actuatedcontroller settings, etc.) can now be optimized using the SA-basedmethod described earlier. However, offsets and phasing sequence can beoptimized using the SA-based method as well.

Regarding output parameters, engineers can still select their desiredperformance measures on the Output Parameters screen, but (similar toSA) it might be easier in many cases to enter Measured values thatactually match the Simulated values. This way, the overall percentdifference, a.k.a., difference value, (between field-measured values andsimulated values) will be displayed as 0.0% for existing conditions,prior to optimization. Because the primary purpose of optimization is tominimize or maximize certain performance measures, matching thefield-measured performance measures becomes a lower priority duringoptimization. That said, matching field-measured and simulatedperformance (i.e., calibration) is certainly recommended before pursuingany optimization. Indeed, confirming that the underlying simulation isrealistic should be a pre-requisite to most optimization efforts.

The prior section “Database of Input Parameters” provided an example onhow to customize the input parameter database. Regarding optimization,the applicable input parameters are typically node-specific and/or phasespecific. Surface Node Yield Point (NETSIM) and Freeway Node MeteringHeadway (FRESIM) are included in the default database, butphase-specific inputs may also be added. For example, if the engineerwanted to analyze the impact of detector lengths on phase #8, they couldadd a new text file containing trial values into the input database.Analyzing the impact of detector lengths could probably be consideredeither sensitivity analysis or optimization. Detector length is notincluded in the default database of input parameters because 1) they'reonly applicable to intersection approach links, and might generate fatalerrors when coded for other links, 2) their value can't exceed thelength of the link that they're on, and 3) their absence is useful inproviding examples for database customization.

It can be assumed that the default database of input parameters(typically available under C:\Program Files (x86)\FHWA\TSIS6.3\Database)contains 66 text files, such that phase #8 detector length will be textfile #67. The CORSIM Reference Manual specifies that detector lengthsare stored on record type 46, columns 24-26. It is important to providetrial values that will not exceed the link length. Sample contents of“67.txt” are shown in FIG. 28. Omitting the “Phase 8” notation wouldcause detector lengths to be modified for all signal phases.

In an earlier section it was mentioned that a pop-up dialog is providedfor Link-Specific input parameters, to allow automated calibration onsome links but not others. For the node-specific input parameters(detector lengths, ramp meter timings, and esoteric actuated controllersettings) typically associated with optimization, a pop-up dialog isagain provided, to allow optimization on selected nodes but not others.

TUTORIAL EXAMPLES Tutorial #1—Surface Street Example (Sample #1.trf)

The first tutorial example is a simple surface-street section with twosignalized intersections. The intersections are separated by 1800 feet.The volume, timing, and laneage details were derived from a 2010 HighwayCapacity Manual example problem, in which access points between theintersections were deleted. The simple link-node geometry is shown inFIG. 10.

Before beginning the normal self-calibration runs, it would be advisableto perform some preliminary random number seed analysis, to determinewhether the simulated performance measures actually representnon-typical results. However, field-measured performance measure valuesmust be entered prior to any type of analysis. The steps to set up apreliminary random number seed analysis are as follows:

-   -   Launch TSIS Next.    -   Open Sample #1.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Next, field-measured values are entered in the Output Parameters screen.The values to be used for this exercise are listed below. These valuescan be entered by clicking on the “Self-Calibration” toolbar icon, andthen switching between links using the combo box; or by right-clickingon specific links in the Map View, and then selecting the“Self-Calibrate” menu item. In each case, after entering the numericvalue, it is then necessary to turn on the checkbox under theSelf-Calibrate? column.

SpeedAverage values for Cumulative Link-Specific Surface (NETSIM)

-   -   Enter 22.96 for links 1--->2 and 2--->1    -   Delay Control per Vehicle values for Cumulative Link-Specific        Surface (NETSIM)    -   Enter 19.30 for links 301--->1 and 402--->2    -   Enter 25.10 for links 2--->1 and 1--->2    -   Enter 39.40 for links 101--->1, 201--->1, 102--->2, and 202--->2

After entering in all of these values, the “total percent difference”should be displayed as 25.1%. Now, the preliminary random number seedanalysis can begin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data row that says Entry Headway Seed, turn on the        checkbox under the Self-Calibrate? column    -   At the far left side of this same data row, select Thorough        under the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 32.1% is displayed, which issignificantly different than the original percent difference (25.1%).When this process is repeated with only the Traffic Stream Seedselected, a Mean Value of 29.2% is displayed, which is alsosignificantly different than the original. When this process is repeatedwith only the Traffic Choice Seed selected, a Mean Value of 27.2% isdisplayed, which is perhaps not significantly different than theoriginal. Finally, Entry Headway Seed and Traffic Stream Seed aresimultaneously selected for Thorough searching (100 total runs, timeestimate 2 minutes 17 seconds).

FIG. 29 shows that data file 34.trf contains a % Difference value thatis closest to the Mean Value for all 100 trial datasets. Therefore, itmight be safer to perform calibration using random number seeds within34.trf, instead of random number seeds within the original dataset.Random number seeds from 34.trf can be automatically imported byclicking on the table cell that says “34.trf”. After this, performancemeasures based on those random number seeds can be automaticallygenerated by performing another CORSIM run, without even leaving theSelf Calibration screen. Just click “Yes” when the program says “Refreshoutputs by performing a CORSIM run?”

Now that the random number seeds have been checked, the self-calibrationprocess can continue. Because NETSIM has a large number of calibrationparameters available for selection, some engineering judgment is neededfor choosing a smaller set of parameters for self-calibration. This isbecause simultaneous calibration of all parameters would result in anunacceptable computer run time. Certain input parameters should beomitted from self-calibration on the basis of traffic networkcharacteristics. For networks with no permissive left-turn movements,the associated input parameters (i.e., Permissive Left-Turn AcceptableGaps, Probability of Left-Turn Lagger, Left-Turn Jumpers) should not bechosen for self-calibration.

The same holds for networks with no queue spillback (Probability ofJoining Spillback), networks with no right-turns-on-red (PermissiveRight-Turn Acceptable Gaps), or networks with no sign-controlledintersections (Near-Side Gap Acceptance at a Sign, Far-Side GapAcceptance at a Sign).

At this stage of the process, research studies (e.g., FHWA-HRT-04-131)for sensitivity analysis could be considered, to help select the mostinfluential input parameters for calibration. For now, a set ofpreliminary self-calibration runs was performed to gauge the effect ofeach global parameter (ignoring link-specific parameters and sensitivityanalysis parameters). The preliminary runs were performed as follows:

-   -   Click on the tab that says Input Parameters and Run Status    -   Choose Vehicle Entry Headway as the only parameter chosen for        self-calibration    -   Set the Searching level to Quick    -   Click on the Generate button    -   Click on the Start button    -   At the end of the run, if the Best Value is lower than the        Original % Difference value by more than 0.1%, make a note of        the Standard Deviation value

This run showed that the trial Vehicle Entry Headways did not produce alower Best Value. Therefore, the Standard Deviation value (0.6) is notrecorded for Vehicle Entry Headway. After a few minutes of preliminaryruns in this manner, there were sixteen input parameters that showedpromise:

17. Maximum Network Initialization Time 1.4 18. Minimum Lane ChangingDeceleration (NETSIM) 1.5 19. Discretionary Lane Change Deceleration(NETSIM) 1.4 20. Lead Vehicle Deceleration (NETSIM) 3.1 21. FollowerVehicle Deceleration (NETSIM) 0.9 22. Driver Aggressiveness Factor(NETSIM) 0.7 23. Safety Factor (NETSIM) 0.7 24. Percentage ofCooperative Drivers (NETSIM) 0.4 25. Lane Change Distance (NETSIM) 0.926. Mean Queue Discharge Headway (NETSIM) 2.5 27. Left-Turn Jumpers(NETSIM) 3.0 28. Turning Speed (NETSIM) 1.4 29. Amber Interval Response(NETSIM) 2.2 30. Free Flow Speed Distribution (NETSIM) 2.9 31. DriverFamiliarity with Paths (NETSIM) 1.6 32. Mean Start-Up Lost Time (NETSIM)1.1

Even at the Quick searching level, simultaneous calibration of allsixteen parameters produces a run time estimate that is consideredunacceptably high. Therefore, these output parameters wereself-calibrated individually at the Thorough level, and the bestsolution was automatically loaded after each run. After a few minutes ofsequential self-calibration runs in this manner, the percent differenceimprovements (original percent difference was 31.9%) were as follows:

17. Maximum Network Initialization Time 28.5 18. Minimum Lane ChangingDeceleration (NETSIM) 28.5 19. Discretionary Lane Change Deceleration(NETSIM) 28.5 20. Lead Vehicle Deceleration (NETSIM) 28.5 21. FollowerVehicle Deceleration (NETSIM) 26.5 22. Driver Aggressiveness Factor(NETSIM) 25.7 23. Safety Factor (NETSIM) 25.7 24. Percentage ofCooperative Drivers (NETSIM) 25.7 25. Lane Change Distance (NETSIM) 25.726. Mean Queue Discharge Headway (NETSIM) 21.9 27. Left-Turn Jumpers(NETSIM) 19.1 28. Turning Speed (NETSIM) 19.1 29. Amber IntervalResponse (NETSIM) 18.7 30. Free Flow Speed Distribution (NETSIM) 16.031. Driver Familiarity with Paths (NETSIM) 15.8 32. Mean Start-Up LostTime (NETSIM) 15.8

At this stage, it may appear that the calibration exercise has doubled(31.9% percent difference, reduced to 15.8%) the accuracy of the model.However, it is still possible that the 15.8% result represents anon-typical result, due to stochastic variation. Thus, it is advisableto perform some “post-calibration” random number seed analysis. Bychoosing Medium searching for all three random number seeds, it ispossible to set up a randomness analysis run with only 125 simulations(run time estimate 2 minutes 40 seconds). After this post-calibrationanalysis run, the Mean Value is displayed as 20.8%, with 33.trf as theMean Dataset. Clicking on 33.trf automatically loads these random numberseeds, and File>Save As can now be used to save this calibrated network.

At this point, the engineer might want to review the calibrated network(animation, inputs, outputs) for possible problems and anomalies. Inaddition, manual (non-automated) “fine-tuning” calibration couldoptionally be performed at this stage.

To summarize the results of this exercise, at the outset, it appearedthat the percentage difference (between simulated and field-measuredperformance) was 25.1%, but randomness analysis revealed the actualdifference was closer to 31.9%. In less than an hour, it was possible toreduce the percentage difference from 32% to around 21%. Since thecalibration of individual settings was done sequentially, instead ofsimultaneously, it is possible that the best combination of settings wasnot yet located. The sequential process was used to keep run times low,but a simultaneous process (with a small number of input parameters)might be preferable. To reduce the percentage difference further below21%, the engineer might want to re-consider the overall list of optionsbelow.

8. Correction of fundamental input data (e.g., volume, timing, laneage)errors 9. Reconciliation of inconsistent performance measuredefinitions* 10. Collection of more accurate field-measured values 11.Elimination of simulation software limitations* 12. Correction ofsimulation software bugs* 13. Manual (non-automated) calibration 14.Automated self-calibration *usually handled by the simulation softwaredevelopers

In this exercise, all calibration input parameters were treated as “fairgame”, such that changing those parameters would be acceptable to theclient. In a real-world project, the engineer would use judgment toavoid self-calibration of 1) parameters the client does not wantchanged, and 2) parameters that are not applicable (e.g., Probability ofJoining Spillback in an undersaturated network) to the given network.Self-calibration of non-applicable parameters would not interfere withaccurate model results; but it would waste time, because non-applicableparameters have no impact on model results.

Tutorial #2—Multi-Period Freeway Example (Sample #2.trf)

The second tutorial example is an extended freeway facility with elevensegments, and five time periods. Volume and laneage details were derivedfrom a 2010 Highway Capacity Manual example problem. The link-nodegeometry is shown in FIG. 12.

Before beginning the normal self-calibration runs, it would be advisableto perform some preliminary random number seed analysis, to determinewhether the simulated performance measures actually representnon-typical results. However, field-measured performance measure valuesmust be entered prior to any type of analysis. The steps to set up apreliminary random number seed analysis are as follows:

-   -   Launch TSIS Next.    -   Open Sample #2.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Next, field-measured values are entered in the Output Parameters screen.The values to be used for this exercise are listed below. These valuescan be entered by clicking on the “Self-Calibration” toolbar icon, andthen switching between links using the combo box; or by right-clickingon specific links in the Map View, and then selecting the“Self-Calibrate” menu item. In each case, after entering the numericvalue, it is then necessary to turn on the checkbox under theSelf-Calibrate? column.

DensityPerLane values for Time Period 3 Link-Specific Freeway (FRESIM)

-   -   Enter 29.30 for link 101--->201    -   Enter 34.20 for link 301--->402    -   Enter 31.90 for link 501--->602    -   Enter 35.80 for link 701--->801    -   Enter 37.70 for link 1101--->1102

SpeedAverage values for Time Period 3 Link-Specific Freeway (FRESIM)

-   -   Enter 59.40 for link 101--->201    -   Enter 57.10 for link 301--->402    -   Enter 58.30 for link 501--->602    -   Enter 56.10 for link 701--->801    -   Enter 55.00 for link 1101--->1102

After entering in all of these values, the “total percent difference”should be displayed as 3.4%. Now, the preliminary random number seedanalysis can begin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data row that says Entry Headway Seed, turn on the        checkbox under the Self-Calibrate? column    -   At the far left side of this same data row, select Medium under        the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 3.6% is displayed, which isslightly higher than the original percent difference. When this processis repeated with only the Traffic Stream Seed selected, the Mean Valuematches the original 3.4%. When this process is repeated with only theTraffic Choice Seed selected, a Mean Value of 3.4% is again displayed.Finally, Entry Headway Seed is selected for Thorough searching (10 runs,time estimate 2 minutes 25 seconds). Afterwards, the screen shows thatdata file 10.trf contains a % Difference value that is equal to the MeanValue (3.7%) for all 10 trial datasets. Therefore, it might be safer toperform calibration using random number seeds within 10.trf, instead ofrandom number seeds within the original dataset. Random number seedsfrom 10.trf can be automatically imported by clicking on the table cellthat says “10.trf”. After this, performance measures based on thoserandom number seeds can be automatically generated by performing anotherCORSIM run, without even leaving the Self Calibration screen. Just click“Yes” when the program says “Refresh outputs by performing a CORSIMrun?”

Now that the random number seeds have been checked, the self-calibrationprocess can continue. Because FRESIM has a large number of calibrationparameters available for selection, some engineering judgment is neededfor choosing a smaller set of parameters for self-calibration. This isbecause simultaneous calibration of all parameters would result in anunacceptable computer run time. Research studies (e.g., FHWA-HRT-04-131)for sensitivity analysis could be considered, to help select the mostinfluential input parameters for calibration. For this exercise, outputparameters were self-calibrated individually and sequentially at theThorough level. Sensitivity analysis parameters (e.g., Entry NodeVolume, Desired Free-Flow Speed, Percent Trucks, etc.) and link-specificparameters were ignored. The best solution was automatically loadedafter each run. After a few minutes of sequential self-calibration runsin this manner, the percent difference improvements (original percentdifference was 3.7%) were as follows:

13. Vehicle Entry Headway 2.9 14. Maximum Network Initialization Time2.9 15. Car Following Sensitivity Multiplier (FRESIM) 2.7 16. CarFollowing Sensitivity (FRESIM) 2.1 17. Time to Complete a Lane Change(FRESIM) 2.1 18. Minimum Entry Headway (FRESIM) 2.1 19. Percentage ofCooperative Drivers (FRESIM) 2.1 20. Lane Change Desire (FRESIM) 2.1 21.Lane Change Advantage (FRESIM) 2.0 22. Maximum Non-EmergencyDeceleration (FRESIM) 2.0 23. Maximum Perceived Deceleration (FRESIM)2.0 24. On-Ramp Anticipatory Lane Change Speed (FRESIM) 2.0 25. FreeFlow Speed Distribution (FRESIM) 2.0 26. On-Ramp Anticipatory LaneChange Distance (FRESIM) 2.0 27. Off-Ramp Reaction Distance (FRESIM) 2.0

At this stage, it may appear that the calibration exercise has nearlydoubled (3.7% percent difference, reduced to 2.0%) the accuracy of themodel. However, it is still possible that the 2.0% result represents anon-typical result, due to stochastic variation. Thus, it is advisableto perform some “post-calibration” random number seed analysis. Bychoosing Medium searching for the Entry Headway and Traffic Choiceseeds, it is possible to set up a randomness analysis run with 25simulations (run time estimate 5 minutes 51 seconds). After thispost-calibration analysis run, the Mean Value is displayed as 2.4%, with1.trf as the Mean Dataset. Clicking on 1.trf automatically loads theserandom number seeds; and File>Save As can now be used to save a copy ofthis calibrated network, under a new name.

At this point, the engineer might want to review the calibrated network(animation, inputs, outputs) for possible problems and anomalies. Inaddition, manual (non-automated) “fine-tuning” calibration couldoptionally be performed at this stage.

To summarize the results of this exercise, at the outset, it appearedthat the percentage difference (between simulated and field-measuredperformance) was 3.7%, and random number analysis confirmed this to be atypical result for various seeds. In less than an hour, it was possibleto reduce the percentage difference from 3.7% to 2.4%. Since thecalibration of individual settings was done sequentially, instead ofsimultaneously, it is possible that the best combination of settings wasnot yet located. The sequential process was used to keep run times low,but a simultaneous process (with a small number of input parameters)might be preferable. To reduce the percentage difference further below2.4%, the engineer might want to re-consider the overall list of optionsbelow.

8. Correction of fundamental input data (e.g., volume, timing, laneage)errors 9. Reconciliation of inconsistent performance measuredefinitions* 10. Collection of more accurate field-measured values 11.Elimination of simulation software limitations* 12. Correction ofsimulation software bugs* 13. Manual (non-automated) calibration 14.Automated self-calibration *usually handled by the simulation softwaredevelopers

In this exercise, all calibration input parameters were treated as “fairgame”, such that changing those parameters would be acceptable to theclient. In a real-world project, the engineer would use judgment toavoid self-calibration of 1) parameters the client does not wantchanged, and 2) parameters that are not applicable to the given network.Self-calibration of non-applicable parameters would not interfere withaccurate model results; but it would waste time, because non-applicableparameters have no impact on model results.

Tutorial #3—Link-Specific Inputs, Surveillance Detector Outputs(corsm1.trf)

The third tutorial example was derived from the “Surface and FreewayDemo” example network, from TSIS. This dataset specifies a single15-minute time period, with 9 surveillance detectors available on threefreeway links (2->51, 510->52, 57->56). The link-node geometry is shownin FIG. 30.

Before beginning the normal self-calibration runs, it would be advisableto perform some preliminary random number seed analysis, to determinewhether the simulated performance measures actually representnon-typical results. However, field-measured performance measure valuesmust be entered prior to any type of analysis. The steps to set up apreliminary random number seed analysis are as follows:

-   -   Launch TSIS Next.    -   Open corsm1.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Next, field-measured values are entered in the Output Parameters screen.The values to be used for this exercise are listed below. These valuescan be entered by clicking on the “Self-Calibration” toolbar icon, andthen switching between links using the combo box; or by right-clickingon specific links in the Map View, and then selecting the“Self-Calibrate” menu item. In each case, after entering the numericvalue, it is then necessary to turn on the checkbox under theSelf-Calibrate? column.

Values for link 2--->51 Time Period 1 Link-Specific Freeway (FRESIM)

-   -   Enter 55.00 for SpeedAverage ID=1 Station=1    -   Enter 800.00 for Volume ID=1 Station=1

After entering in these values, the “total percent difference” should bedisplayed as 7.5%. Now, the preliminary random number seed analysis canbegin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data row that says Entry Headway Seed, turn on the        checkbox under the Self-Calibrate? column    -   At the far left side of this same data row, select Medium under        the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 7.5% is displayed, which isidentical to the original percent difference. When this process isrepeated with only the Traffic Stream Seed selected, a Mean Value of10.2% is generated. When this process is repeated with only the TrafficChoice Seed selected, a Mean Value of 6.9% is generated. Finally, theTraffic Stream and Traffic Choice seeds are selected simultaneously (25runs˜2 minutes), producing a Mean Value of 11.1%. Because the 11.1%difference appears to be more typical than the original 7.5%, randomnumber seeds should be imported by clicking on the Mean Dataset (1.trf),which produces the closest (10.8%) difference to 11.1%. During thisimport process, click “Yes” when asked “Refresh outputs by performing aCORSIM run?”

Now that the random number seeds have been checked, the self-calibrationprocess can continue. In this example, assume that the client is willingto calibrate only four input parameters (car-following multiplier,warning sign distance, anticipatory lane change speed, anticipatory lanechange distance) on a link-specific (not global) basis. Let's start bycalibrating the car-following multiplier on link 2->51. First, unselectthe Self-Calibrate checkbox for both random number seed inputparameters. Then, select Thorough searching for Link-Specific CarFollowing Multiplier (FRESIM). When the Self-Calibrate checkbox isturned on for this input parameter, the Link-Specific Input ParameterCalibration dialog will pop up. Click on Unselect All before selectinglink 2->51, as illustrated in FIG. 31.

Click OK to save link-specific calibration for link 2->51. Afterclicking on Generate, and then clicking on Start, the Best Value is now7.9% for all seven trial values of car-following multiplier. Therefore,the value of car-following multiplier producing 7.9% should be importedat this time, from dataset 1.trf. When Link-Specific On-Ramp Lane ChangeSpeed (FRESIM) is self-calibrated in this same manner, the Best Value isnow 6.3%, so this lane change speed should be imported from dataset1.trf. When Link-Specific On-Ramp Lane Change Distance (FRESIM) isself-calibrated in this same manner, the Best Value is now 4.8%, so thislane change distance should be imported from dataset 1.trf. WhenLink-Specific Off-Ramp Reaction Distance (FRESIM) is self-calibrated inthis same manner, the Best Value remains 4.8%.

The next step is to check the imported calibration settings acrossdifferent random number seeds. After unchecking Self-Calibrate for theOff-Ramp Reaction Distance, re-check Self-Calibrate for the TrafficChoice and Traffic Stream seeds, with Medium searching (25 runs ˜2minutes). After clicking on Generate and then Start, to analyze theserandom number seeds, the Mean Value is 6.8% for dataset 13.trf.Therefore, the final result of this exercise was an improvement (insurveillance detector output discrepancies) from 10.8% to 6.8%.

Note that the time estimate for calibrating all four input parameterssimultaneously was estimated as 15 hours on the computer used for thisexercise. If the analyst were willing to let the program run for 15hours, it is likely that a better combination of inputs would be found.

Tutorial #4—Simple Sensitivity Analysis (actctrl.trf)

The fourth tutorial example was derived from the “Actuated Control Demo”example network, from ISIS. This dataset specifies two 5-minute timeperiods, with the link-node geometry shown in FIG. 32.

As with the previous tutorial exercises, a single preliminary simulationrun is needed to enable the self-calibration feature, which will be usedfor sensitivity analysis:

-   -   Launch TSIS Next.    -   Open actctrl.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Before beginning the sensitivity analysis (SA) runs, preliminary randomnumber seed analysis is again recommended, to determine whethersimulated performance measures represent typical results. Forself-calibration, field-measured performance measure values wouldnormally be entered at this stage. For an application of SA as opposedto calibration, time can be saved during this step by setting one of themeasured values equal to its simulated value. This value can be enteredby clicking on the “Self-Calibration” toolbar icon, and then choosingyour desired performance measure on the Output Parameters screen; or byright-clicking on specific links in the Map View, and then selecting the“Self-Calibrate” menu item. The screen shown in FIG. 33 illustratessetting the measured SpeedAverage (Cumulative, Global) equal to itssimulated value (17.64):

After entering in this value, the “total percent difference” should bedisplayed as 0.0%. Now, the preliminary random number seed analysis canbegin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data rows for Entry Headway Seed, Traffic Stream Seed,        and Traffic Choice Seed, turn on the checkbox under the        Self-Calibrate? column    -   At the far left side of this same data row, select Quick under        the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 1.6% is displayed. Because the1.6% difference appears to be more typical than the original 0.0%,random number seeds should be imported by clicking on the Mean Dataset(4.trf). During this import process, click “Yes” when asked “Refreshoutputs by performing a CORSIM run?”

Now that the random number seeds have been checked, the sensitivityanalysis (SA) process can continue. In this example, an analysis will bedone on the impact of Percent Trucks on SpeedAverage (Cumulative,Global). First, unselect the Self-Calibrate checkbox for all threerandom number seed input parameters. Then, select Medium searching forEntry Node Percent Trucks. After clicking on Generate, and then clickingon Start, the Best Value is naturally still 1.6% once the simulationruns are completed. Since it is not necessary to be concerned with SAthan calibration, click on the Output Parameters tab. Once inside theOutput Parameters screen, click on the radio buttons for Cumulative andGlobal, and click on the S.A. button at the bottom of the screen. Thetable of SA results illustrated in FIG. 34 shows that traffic networkperformance generally deteriorates as the percentage of trucksincreases. Note that the S.A. button (Output Parameters screen) onlybecomes enabled and visible after clicking on Generate (Input Parametersand Run Status screen).

Tutorial #5—Simple Optimization (Sample #1.trf)

The fifth tutorial example involves the same pair of signalizedintersections from Example #1. As with the previous tutorial exercises,a single preliminary simulation run is needed to enable theself-calibration feature, which will be used for sensitivity analysis:

-   -   Launch TSIS Next.    -   Open actctrl.trf.    -   Turn on the following checkbox: Options>Preferences>Output        Files>After CORSIM runs, copy OUT/CSV files to the TRF file        folder.    -   Perform one preliminary CORSIM simulation run by clicking on the        “Run CORSIM” toolbar icon.

Before beginning the optimization runs, preliminary random number seedanalysis is again recommended, to determine whether simulatedperformance measures represent typical results. For self-calibration,field-measured performance measure values would normally be entered atthis stage. For an application of optimization as opposed tocalibration, time can be saved during this step by setting one of themeasured values equal to its simulated value. This value can be enteredby clicking on the “Self-Calibration” toolbar icon, and then choosingyour desired performance measure on the Output Parameters screen; or byright-clicking on specific links in the Map View, and then selecting the“Self-Calibrate” menu item. The screen in FIG. 35 illustrates settingthe measured Delay Control Per Vehicle (Cumulative, Link-Specific) equalto its simulated value (32.01), on link 1-->2:

After entering in this value, the “total percent difference” should bedisplayed as 0.0%. Now, the preliminary random number seed analysis canbegin.

-   -   Click on the tab that says Input Parameters and Run Status    -   In the data rows for Entry Headway Seed, Traffic Stream Seed,        and Traffic Choice Seed, turn on the checkbox under the        Self-Calibrate? column    -   At the far left side of this same data row, select Quick under        the Searching column    -   Click on the Generate button    -   Click on the Start button

At the end of the run, a Mean Value of 10.7% is displayed. Because the10.7% difference appears to be more typical than the original 0.0%,random number seeds should be imported by clicking on the Mean Dataset(7.trf). During this import process, click “Yes” when asked “Refreshoutputs by performing a CORSIM run?”

Now that the random number seeds have been checked, the optimizationprocess can continue. In this example the Yield Point at node #2 will beoptimized, with Delay Control Per Vehicle (Cumulative, Link-Specific) atlink 1-->2 as the optimization objective. First, unselect theSelf-Calibrate checkbox for all three random number seed inputparameters. Then, select Medium searching for Surface Node Yield Point(NETSIM). After this the pop-up dialog should appear, as shown in FIG.36.

After clicking on Generate, and then clicking on Start, the Best Valueappears to be 43.1% once the simulation runs are completed. Since it isnot necessary to be more concerned with optimization than calibration,click on the Output Parameters tab. Once inside the Output Parametersscreen, click on the radio buttons for Cumulative and Link-Specific(link 1--->2), and click on the S.A. button at the bottom of the screen.The table of optimization results illustrated in FIG. 37 shows thatcontrol delay is minimized at a yield point of 25 seconds. Note that theS.A. button (Output Parameters screen) only becomes enabled and visibleafter clicking on Generate (Input Parameters and Run Status screen).

Although Surface Node Yield Point is present within the default databaseof input parameters, the default trial values are very low, to avoidexceeding typical cycle lengths. In most cases the database file shouldbe edited, so that higher yield point values can be examined, but thehigher yield point values should never exceed the cycle length. AlthoughDelay Control Per Vehicle can be viewed on the same screen as the yieldpoint values, other performance measures can only be viewed by scrollingdown the list. In cases such as this it might be preferable tocopy-and-paste the tabular outputs into a spreadsheet file, alsoillustrated in FIG. 38, so that all desired outputs can besimultaneously visible.

Summary

The existing, non-automated methods of calibration have been considereddifficult and/or inadequate by leading experts and practitioners. Therehas been a significant amount of research in the area of automatedcalibration techniques, but many of these projects have not provided theflexibility and practicality typically required by real-world engineers.With this in mind, the self-calibration features within TSIS-CORSIM weredesigned with an eye on maximizing practicality, flexibility, andease-of-use. The implemented methodology allows engineers to quickly andeasily select a set of input and output parameters for calibration. Thismethodology also allows engineers to prioritize specific input andoutput parameters, and specify their tolerable computer run time, priorto initiating the self-calibration process. The directed brute forcesearch process is thought to be a key element in making this methodologyflexible and practical, for real-world use. The self-calibrationfeatures can also facilitate sensitivity analysis and optimization for awide variety of input and output parameters, helping practitioners andresearchers to gain a better understanding of the modeling process.

By providing simple lists of input and output parameters to choose from,the software is designed to make calibration easier, and reduce theamount of engineering expertise required. By providing optional priorityweighting for each output parameter, allowing the user to select searchthoroughness (Quick, Medium, Thorough) for each input parameter, andallowing the user to modify the database of input parameters if desired,the software is designed to provide more flexibility than previousautomated calibration techniques. By automatically generating all of thetrial input data, and all of the trial data sets, the software isdesigned to reduce the amount of time and money typically spent oncalibration. By automatically displaying recent simulation results nextto the field-measured results, and by providing run time estimates inadvance, the software is designed to make the process of calibrationmore simple and practical. In essence, the software is intended to makecalibration easier, thus improving the accuracy of simulation.

Largely due to computer speed limitations, it is believed that automatedcalibration processes cannot fully replace engineering judgment,engineering expertise, or manual (non-automated) calibration. Howeverthe same could be said regarding many popular engineering softwaretools, all of which can lead to incorrect engineering decisions when notapplied properly. The automated tools also cannot defend againstfundamental (volume, timing, laneage) input data errors, simulationsoftware bugs/limitations, or inconsistent performance measuredefinitions. Despite this, these software tools can hopefully “bridgethe gap”, in terms of significantly reducing the amount of time andexpertise required for complex engineering projects.

It should be appreciated that the software components described hereinmay, when loaded into a central processing unit (CPU) and executed,transform the CPU and the overall computer architecture from ageneral-purpose computing system into a special-purpose computing systemcustomized to facilitate the functionality presented herein. The CPU maybe constructed from any number of transistors or other discrete circuitelements, which may individually or collectively assume any number ofstates. More specifically, the CPU may operate as a finite-statemachine, in response to executable instructions contained within thesoftware modules disclosed herein. These computer-executableinstructions may transform the CPU by specifying how the CPU transitionsbetween states, thereby transforming the transistors or other discretehardware elements constituting the CPU.

Certain techniques set forth herein may be described or implemented inthe general context of computer-executable instructions, such as programmodules, executed by one or more computing devices. Generally, programmodules include routines, programs, objects, components, and datastructures that perform particular tasks or implement particularabstract data types.

Embodiments may be implemented as a computer process, a computingsystem, or as an article of manufacture, such as a computer programproduct or computer-readable medium. Certain methods and processesdescribed herein can be embodied as code and/or data, which may bestored on one or more computer-readable media. Certain embodiments ofthe invention contemplate the use of a machine in the form of a computersystem within which a set of instructions, when executed, can cause thesystem to perform any one or more of the methodologies discussed above.Certain computer program products may be one or more computer-readablestorage media readable by a computer system and encoding a computerprogram of instructions for executing a computer process.

Computer-readable media can be any available computer-readable storagemedia or communication media that can be accessed by the computersystem.

Communication media include the mechanisms by which a communicationsignal containing, for example, computer-readable instructions, datastructures, program modules, or other data, is transmitted from onesystem to another system. The communication media can include guidedtransmission media, such as cables and wires (e.g., fiber optic,coaxial, and the like), and wireless (unguided transmission) media, suchas acoustic, electromagnetic, RF, microwave and infrared, that canpropagate energy waves.

By way of example, and not limitation, computer-readable storage mediamay include volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. For example, a computer-readable storage medium can includevolatile memory such as random access memories (RAM, DRAM, SRAM); andnon-volatile memory such as flash memory, various read-only-memories(ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic/ferroelectricmemories (MRAM, FeRAM), and magnetic and optical storage devices (harddrives, magnetic tape, CDs, DVDs). “Computer-readable storage media” donot consist of carrier waves or propagating signals.

In addition, the methods and processes described herein can beimplemented in hardware modules. For example, the hardware modules caninclude, but are not limited to, application-specific integrated circuit(ASIC) chips, field programmable gate arrays (FPGAs), and otherprogrammable logic devices now known or later developed. When thehardware modules are activated, the hardware modules perform the methodsand processes included within the hardware modules.

Any reference in this specification to “one embodiment,” “anembodiment,” “example embodiment,” etc., means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the invention. Theappearances of such phrases in various places in the specification arenot necessarily all referring to the same embodiment. In addition, anyelements or limitations of any invention or embodiment thereof disclosedherein can be combined with any and/or all other elements or limitations(individually or in any combination) or any other invention orembodiment thereof disclosed herein, and all such combinations arecontemplated with the scope of the invention without limitation thereto.

All patents, patent applications, provisional applications, and otherpublications referred to or cited herein are incorporated by referencein their entirety, including all figures and tables, to the extent theyare not inconsistent with the explicit teachings of this specification.Additionally, the entire contents of the references cited within thereferences cited herein are also entirely incorporated by reference.

The examples and embodiments described herein are for illustrativepurposes only and that various modifications or changes in light thereofwill be suggested to persons skilled in the art and are to be includedwithin the spirit and purview of this application.

The invention has been described herein in considerable detail, in orderto comply with the Patent Statutes and to provide those skilled in theart with information needed to apply the novel principles, and toconstruct and use such specialized components as are required. However,the invention can be carried out by specifically different equipment anddevices, and that various modifications, both as to equipment detailsand operating procedures can be effected without departing from thescope of the invention itself. Further, although the present inventionhas been described with reference to specific details of certainembodiments thereof and by examples disclosed herein, it is not intendedthat such details should be regarded as limitations upon the scope ofthe invention except as and to the extent that they are included in theaccompanying claims.

What is claimed is:
 1. A computer-implemented interface apparatus for amodel that: provides one or more selectable input parameters through anoutput device; receives a selection of at least one input parameter ofthe one or more selectable input parameters; receives a range of valuesfor use with the selected at least one input parameter; receives, via aninput device from a user, a selection for the model to initiate at leastone of an automated calibration, sensitivity analysis, and optimizationof the selected at least one input parameter; archives, during executionof the model, a dynamically updated list of one or more completed trialsfor the selected at least one input parameter; and provides an optionfor displaying at least one of the archived one or more completedtrials.
 2. The computer-implemented interface apparatus according toclaim 1, that further: in response to receiving the selection for themodel to conduct automated calibration, archives an associateddifference value with each of the one or more completed trials, whereinthe difference value is generated based on the difference between a oneor more model-calculated value and a one or more ground truth value forthe selected at least one input parameter; and, provides an option fordisplaying the associated difference value with the one or morecompleted trials.
 3. The computer-implemented interface apparatusaccording to claim 1, that further: in response to receiving theselection for the model to conduct sensitivity analysis, archives anassociated set of model generated output parameter values for each ofthe one or more completed trials; and, provides an option for displayingthe model generated output parameter values with the one or morecompleted trials.
 4. The computer-implemented interface apparatusaccording to claim 1, that further: in response to receiving theselection for the model to conduct optimization of the selected at leastone input parameter, archives an associated difference value for each ofthe one or more completed trials, wherein the difference value isgenerated based on the difference between a one or more model-calculatedvalue and a one or more objective function value for the selected atleast one input parameter; and provides an option for displaying theassociated difference value with the one or more completed trials.
 5. Acomputer-implemented method for automated calibration of a model, themethod comprising: providing one or more selectable input parametersthrough an output device; receiving a selection of at least one inputparameter of the one or more selectable input parameters via an inputdevice from a user; receiving a range of values for use with theselected at least one input parameter; in response to receiving aselection for the model to initiate at least one of automatedcalibration, sensitivity analysis, and optimization of the selected atleast one input parameter, executing one or more trials utilizing therange of values, whereby the model evaluates, for each trial, a trialvalue for each selected at least one input parameter; and, archivingduring execution of the one or more trials, a dynamically updated listof one or more completed trials.
 6. The computer-implemented methodaccording to claim 5, further comprising: in response to receiving theselection for the model to conduct automated calibration, generating anassociated difference value for each trials, wherein the differencevalue is generated based on the difference between a one or moremodel-calculated value and a one or more ground truth value for theselected at least one input parameter; and archiving the associateddifference value with each completed trial.
 7. The computer-implementedmethod according to claim 5, further comprising: in response toreceiving the selection for the model to initiate sensitivity analysis,generating an at least one model-generated associated output parametervalue for each trial; and archiving the associated output parameter witheach completed trial.
 8. The computer-implemented method according toclaim 5, further comprising: in response to receiving the selection forthe model to initiate optimization, generating an at least oneassociated difference value for each trial, wherein the difference valueis generated based on the difference between an at least onemodel-calculated value and an at least one objective function value forthe selected at least one input parameter; and archiving the associateddifference value with each completed trial.